Transcript
Page 1: How to integrate the Elasticsearch Logstash Kibana (ELK ... · How to integrate the Elasticsearch Logstash Kibana (ELK) log analytics stack into IBM Bluemix Nick Cawood IBM Cloud

How to integrate the Elasticsearch Logstash

Kibana (ELK) log analytics stack into IBM Bluemix

Nick Cawood IBM Cloud Client Adoption and Technical Enablement

Client and Technical Engagement13 August 2016

Page of 1 71

CONTENTS13

Introducon13 313

What13 is13 Elascsearch13 Logstash13 Kibana13 313

LOGSTASH 3 ELASTICSEARCH 4 KIBANA 4 ELK SYSTEM ARCHITECTURE 4

Service13 Management13 architecture13 613

Log13 Monitoring13 System13 architecture13 613

Bluemix13 environment13 system13 architecture13 613

Creang13 a13 virtual13 server13 in13 Bluemix13 713

CREATE AN SSH KEY 8 PROVISIONING A VIRTUAL SERVER 9 SET UP THE VIRTUAL SERVER 12

Log on to the virtual server 12 Acces the root user 12 Copy files to the virtual server 13 Install and access the virtual server via VNC server 13

Install13 the13 ELK13 stack13 1313

INSTALL ELASTICSEARCH 14 Start Elasticsearch 14

INSTALL KIBANA 15 Start Kibana 18

BROADCAST BLUEMIX APP LOG RECORDS 19 INSTALL LOGSTASH 19

Write Logstash configuration 19 Start Logstash 21 Monitor Logstash processing 21

Use13 ELK13 2213

KIBANA DISCOVER PAGE 22 Change the time filter 22 Custom search queries 23 Saved searches 24

BUILD SIMPLE KIBANA HISTOGRAMS 27 Alerng13 with13 ELK13 3613

CREATE ALERTS IN LOGSTASH 36 ENRICH ALERTS WITH MESSAGE DETAIL 38

Grok 38 Translate 38

ADD CUSTOM PLUGINS TO LOGSTASH 39 Filter translate plugin 39 Output Slack plugin 39 Output syslog plugin 40

INTEGRATE LOGSTASH WITH SLACK 40 INTEGRATE LOGSTASH WITH PAGERDUTY 45 INTEGRATE LOGSTASH WITH IBM NETCOOL OMNIBUS 48

Appendix13 Logstash13 configuraon13 51

Page of 2 71

Introduc0on13 IBMreg13 Bluemixreg13 IBMrsquos13 PlaRorm13 as13 a13 Service13 (PaaS)13 allows13 you13 to13 rapidly13 build13 and13 deploy13 an13 applicaon13 Applicaons13 can13 run13 in13 a13 nave13 or13 a13 hybrid13 Bluemix13 environment13 13 13

This13 document13 describes13 the13 integraon13 of13 the13 Elascsearch13 Logstash13 Kibana13 (ELK)13 log13 analycs13 open13 source13 stack13 into13 the13 Bluemix13 environment13 how13 to13 collect13 logs13 from13 Bluemix13 applicaons13 (apps)13 and13 send13 alerts13 from13 those13 log13 records13 to13 selected13 alertevent13 noficaon13 targets13 (PagerDuty13 Slack13 and13 IBM13 Netcool13 Omnibus)13

This13 document13 describes13 how13 to13 provision13 Bluemix13 virtual13 servers13 in13 the13 Bluemix13 environment13 install13 the13 ELK13 stack13 on13 a13 Bluemix13 virtual13 server13 and13 configure13 the13 ELK13 stack13 to13 integrate13 with13 other13 components13 deployed13 on13 the13 Bluemix13 environment13

The13 steps13 to13 integrate13 the13 ELK13 stack13 into13 the13 Cloud13 Architecture13 and13 Soluon13 Engineering13 (CASE)13 environment13 are13 used13 as13 the13 basis13 for13 this13 document13 13 For13 reference13 both13 the13 ELK13 architecture13 within13 the13 CASE13 environment13 and13 the13 whole13 CASE13 environment13 are13 defined13 using13 system13 architecture13 diagrams13 as13 well13 as13 diagrams13 defining13 the13 ELK13 structure13 and13 ELK13 integraon13 into13 Bluemix13 architecture13

What13 is13 Elas0csearch13 Logstash13 Kibana13 Elascsearch13 Logstash13 Kibana13 is13 an13 open13 source13 stack13 of13 three13 applicaons13 that13 work13 together13 to13 provide13 an13 end-shy‐to-shy‐end13 search13 and13 visualisaon13 plaRorm13 for13 the13 invesgaon13 of13 log13 file13 sources13 in13 real13 me13 Log13 file13 records13 can13 be13 searched13 from13 a13 variety13 of13 log13 sources13 charts13 and13 dashboards13 built13 to13 visualize13 the13 log13 records13 The13 three13 applicaons13 are13 Elascsearch13 Logstash13 and13 Kibana13

Logstash13 Logstash13 is13 an13 applicaon13 that13 allows13 the13 ELK13 stack13 to13 manage13 the13 gathering13 transformaon13 and13 transport13 of13 log13 file13 records13 Logstash13 is13 able13 to13 pull13 records13 from13 remote13 sources13 or13 listen13 for13 records13 on13 specified13 ports13

Logstash13 has13 three13 areas13 of13 funcon13

bull Process13 input13 of13 the13 log13 file13 records13 from13 the13 remote13 source13 You13 can13 configure13 Logstash13 to13 receive13 records13 in13 a13 variety13 of13 methods13 for13 example13 reading13 from13 a13 flat13 file13 or13 from13 a13 TCP13 port13 or13 direct13 from13 a13 remote13 or13 local13 syslog13 These13 methods13 are13 defined13 in13 the13 Logstash13 online13 support13 and13 input13 secon13 of13 the13 Logstash13 conf13 file13 Once13 all13 input13 processing13 is13 completed13 Logstash13 moves13 onto13 the13 next13 funcon13

bull Transform13 and13 enrich13 the13 log13 records13 that13 take13 place13 in13 the13 Logstash13 filter13 secon13 Any13 number13 of13 changes13 to13 the13 format13 (and13 content13 if13 required)13 of13 the13 log13 file13 record13 can13 be13 made13 using13 the13 various13 methods13 provided13 with13 Logstash13 for13 example13 methods13 to13 add13 and13 enrich13 log13 record13 fields13 and13 the13 flagging13 I13 describe13 excluding13 and13 liming13 records13 for13 alerng13 later13 in13 this13 document13 Aaer13 the13 log13 record13 is13 transformed13 Logstash13 processes13 the13 record13

bull Send13 the13 log13 records13 to13 the13 Elascsearch13 applicaon13 from13 the13 output13 secon13 of13 the13 Logstash13 configuraon13 A13 specific13 elascsearch13 method13 is13 included13 with13 the13 Logstash13 instance13 that13 sends13 the13 log13 records13 to13 the13 Elascsearch13 instance13 However13 as13 with13 the13 input13 and13 filter13

Page of 3 71 Page of 3 71

secons13 a13 variety13 of13 methods13 are13 available13 to13 process13 the13 data13 sent13 from13 the13 filter13 secon13 For13 example13 the13 records13 can13 be13 sent13 to13 external13 alerng13 tools13 such13 as13 PagerDuty13 social13 media13 tools13 such13 as13 Slack13 or13 event13 management13 hubs13 like13 IBM13 Netcool13 Omnibus13 via13 output13 methods13 syslog13 or13 hcp13 13

As13 an13 open13 source13 tool13 non-shy‐Logstash13 employees13 develop13 other13 methods13 and13 share13 them13 with13 the13 user13 community13 for13 addion13 to13 the13 library13 of13 methods13 Logstash13 can13 execute13 (examples13 of13 this13 process13 are13 included13 later13 in13 this13 document)13

Elas0csearch13 Elascsearch13 is13 the13 index13 store13 and13 query13 applicaon13 on13 the13 ELK13 stack13 It13 provides13 a13 standard13 full13 text13 search13 facility13 of13 the13 log13 file13 records13 processed13 by13 Logstash13 The13 ELK13 online13 help13 documentaon13 describes13 Elascsearch13 as13 follows13 ldquoIt13 is13 generally13 used13 as13 the13 underlying13 enginetechnology13 that13 powers13 applicaons13 that13 have13 complex13 search13 features13 and13 requirementsrdquo13

Kibana13 Kibana13 is13 the13 user13 interface13 for13 the13 ELK13 stack13 Kibana13 integrates13 with13 Elascsearch13 to13 visualise13 the13 results13 of13 searches13 and13 allow13 the13 creaon13 of13 charts13 and13 dashboards13 based13 on13 the13 searches13 The13 searches13 charts13 and13 dashboard13 can13 visualise13 real-shy‐me13 or13 historical13 log13 records13 Users13 access13 Kibana13 through13 an13 hcp13 URL13 from13 a13 client13 web13 browser13

ELK13 System13 Architecture13 The13 ELK13 stack13 can13 be13 described13 as13 follows13

Page of 4 71 Page of 4 71

13

Page of 5 71 Page of 5 71

Service13 Management13 architecture13 The13 IBM13 Cloud13 Architecture13 Center13 created13 a13 standard13 reference13 architecture13 for13 how13 to13 set13 up13 your13 cloud13 environemnt13 to13 handle13 incident13 management13 13

View13 the13 standard13 Service13 Management13 architecure13

13 13

Log13 Monitoring13 System13 architecture13 The13 following13 diagram13 shows13 a13 standard13 architecture13 for13 the13 log-shy‐monitoring13 system13 described13 in13 this13 tutorial13 13

13

Bluemix13 environment13 system13 architecture13 The13 ELK13 stack13 can13 be13 installed13 into13 a13 variety13 of13 environments13 When13 using13 Bluemix13 the13 following13 figure13 shows13 what13 the13 environment13 might13 look13 like13

Page of 6 71 Page of 6 71

13

The13 Bluemix13 cloud13 plaRorm13 offers13 a13 Public13 network13 and13 access13 to13 US13 South13 and13 UK13 environments13 From13 all13 these13 environments13 third-shy‐party13 cloud-shy‐based13 soluons13 are13 also13 accessible13

The13 ELK13 stack13 is13 installed13 in13 the13 Bluemix13 United13 Kingdom13 environment13 because13 this13 is13 the13 environment13 where13 Bluemix13 has13 virtual13 servers13 available13 for13 provisioning13 Other13 applicaons13 in13 the13 CASE13 project13 were13 also13 installed13 on13 virtual13 servers13 in13 the13 Bluemix13 United13 Kingdom13 region13

Bluemix13 apps13 were13 used13 to13 create13 a13 system13 to13 be13 monitored13 by13 ELK13 These13 apps13 were13 hosted13 in13 the13 Bluemix13 US13 South13 environment13 using13 Bluemix13 microservices13 which13 produce13 logs13 during13 normal13 operaon13 and13 during13 crashes13 and13 outages13

Through13 the13 Bluemix13 Cloud13 Foundry13 process13 the13 log13 records13 can13 be13 broadcast13 from13 the13 Bluemix13 apps13 so13 Logstash13 can13 receive13 them13 and13 send13 them13 to13 Elascsearch13 within13 the13 ELK13 stack13 or13 other13 virtual13 servers13 (hosng13 other13 IBM13 or13 third-shy‐party13 products)13 in13 the13 Bluemix13 United13 Kingdom13 environment13 or13 to13 third-shy‐party13 applicaons13 in13 the13 public13 cloud13 (Pager13 Duty13 Slack13 etc)13

Crea0ng13 a13 virtual13 server13 in13 Bluemix13 With13 Bluemix13 you13 can13 provision13 a13 virtual13 server13 in13 the13 cloud13 to13 install13 standalone13 or13 integrated13 system13 components13 Not13 all13 Bluemix13 regions13 offer13 this13 service13 all13 the13 me13 due13 to13 beta-shy‐project13 constraints13 In13 this13 arclersquos13 examples13 virtual13 servers13 were13 allowed13 to13 be13 provisioned13 in13 the13 United13 Kingdom13 region13 For13 more13 informaon13 read13 how13 to13 provision13 a13 Bluemix13 virtual13 server13 including13 some13 simple13 setup13 steps13 for13 ease13 of13 use13

Page of 7 71 Page of 7 71

Create13 an13 SSH13 key13 To13 access13 your13 virtual13 server13 once13 it13 is13 provisioned13 allocate13 an13 SSH13 key13 to13 the13 virtual13 server13

To13 generate13 a13 publicprivate13 RSA13 key13 pair13 follow13 these13 steps13

1 In13 a13 UNIX13 or13 Linuxreg13 window13 run13 the13 following13 command13 specifying13 a13 ltuniquekeynamegt13 of13 your13 your13 choice13 and13 a13 password13 or13 passphrase13 (memorize13 the13 password)$13 ssh-shy‐keygen13 -shy‐t13 rsa13 -shy‐f13 ltuniquekeynamegt13

2 Enter13 the13 passphrase13 (empty13 for13 no13 passphrase)13

3 Enter13 the13 same13 passphrase13 again13

Your13 idenficaon13 has13 been13 saved13 in13 ltuniquekeynamegt13

Your13 public13 key13 has13 been13 saved13 in13 ltuniquekeynamegtpub13

The13 key13 fingerprint13 is13

d6268464506525930ce4632c693c203c13 ltusergtltservergt13

The13 keys13 randomart13 image13 is13

+--[ RSA 2048]----+

|o ++=o |

| E o ooo |

| |

| + o |

| S o |

| o |

| |

| |

| |

+-----------------+

Two13 keys13 are13 created13 public13 and13 private13 The13 public13 key13 contents13 are13 used13 to13 create13 a13 virtual13 server13 with13 a13 public13 IP13 address13 The13 private13 key13 is13 used13 to13 access13 the13 virtual13 server13 from13 a13 client13 machine13

Page of 8 71 Page of 8 71

Provisioning13 a13 virtual13 server13 1 A13 Bluemix13 account13 is13 required13 for13 this13 process13 Sign13 up13 here13 and13 log13 in13 Make13 sure13 your13

environment13 is13 correct13 For13 example13 if13 you13 are13 in13 the13 United13 Kingdom13 select13 the13 United13 Kingdom13 region13 in13 the13 Bluemix13 console13

13

13

2 Add13 a13 virtual13 server13 to13 your13 Bluemix13 environment13 Click13 Run13 Virtual13 Servers13

13

3 Select13 an13 OS13 image13 from13 the13 EXISTING13 list13 For13 ELK13 select13 Centos13 OS13 (use13 latest13 available13 version)13

Page of 9 71 Page of 9 71

4 Name13 the13 virtual13 server13

5 Make13 sure13 Assign13 public13 IP13 address13 is13 checked13

13

6 Click13 Add13 Key13 to13 add13 the13 SSH13 key13 you13 created13 previously13 Copy13 the13 contents13 of13 the13 public13 key13 into13 the13 Public13 Key13 to13 import13 field13

Page of 10 71 Page of 10 71

13

7 Click13 OK13 Then13 click13 CREATE13

The13 virtual13 server13 is13 provisioned13 and13 added13 to13 the13 Bluemix13 dashboard13 under13 VIRTUAL13 SERVERS13

Page of 11 71 Page of 11 71

13

Set13 up13 the13 virtual13 server13

Log13 on13 to13 the13 virtual13 server13 Now13 you13 can13 log13 on13 to13 the13 virtual13 server13 and13 add13 new13 users13 and13 programs13 to13 make13 it13 easier13 to13 access13 the13 virtual13 server13

To13 access13 the13 server13 go13 to13 the13 UNIX13 or13 Linux13 window13 that13 you13 used13 to13 create13 the13 SSH13 key13 In13 the13 directory13 where13 the13 ltuniquekeynamegt13 files13 exist13 run13 the13 following13 command13 referencing13 the13 SSH13 keyname13 the13 public13 IP13 address13 Bluemix13 allocated13 for13 the13 virtual13 server13 and13 the13 password13 for13 the13 SSH13 key13

[userserver ~]$ ssh -i ltuniquekeynamegt ibmcloudltpublicIPaddressgt

Enter13 the13 passphrase13 for13 the13 key13 ltuniquekeynamegt13

Last login Tue Jul 5 125546 2016 from 108-210-96-231lightspeedrlghncsbcglobalnet

[ibmcloudelkforcase ~]$

ibmcloud13 is13 the13 default13 user13 for13 the13 Bluemix13 virtual13 servers13 and13 allows13 SSH13 to13 access13 the13 server13

Acces13 the13 root13 user13 To13 access13 the13 root13 user13 log13 on13 as13 ibmcloud13 user13 and13 run13 the13 following13 command13

[ibmcloudelkforcase ~]$ sudo bash

[rootelkforcase ibmcloud]

The13 root13 user13 can13 be13 used13 to13 add13 programs13 to13 the13 server13 and13 create13 and13 edit13 users13 as13 required13 to13 access13 applicaons13 installed13 on13 the13 virtual13 server13

Page of 12 71 Page of 12 71

Copy13 files13 to13 the13 virtual13 server13 To13 move13 installaon13 files13 to13 the13 server13 go13 to13 the13 UNIX13 or13 Linux13 window13 you13 used13 to13 create13 the13 SSH13 key13 and13 the13 directory13 where13 the13 ltuniquekeynamegt13 files13 exist13 Move13 the13 installaon13 files13 to13 the13 same13 locaon13

Run13 the13 following13 command13 referencing13 the13 SSH13 keyname13 the13 public13 IP13 address13 Bluemix13 allocated13 for13 the13 virtual13 server13 and13 the13 password13 for13 the13 SSH13 key13

[userserver ~]$ scp ndashi ltuniquekeynamegt ltINSTALLATIONFILENAMEgt ibmcloudltpublicIPaddressgthomeibmcloud

Enter13 the13 passphrase13 for13 key13 cloudkey313

This13 copies13 the13 file13 using13 the13 ibmcloud13 user13 to13 the13 homeibmcloud13 directory13 on13 the13 virtual13 server13

Install13 and13 access13 the13 virtual13 server13 via13 VNC13 server13 Using13 the13 ssh13 command13 to13 access13 the13 virtual13 server13 is13 not13 the13 most13 stable13 method13 and13 does13 not13 offer13 X-shy‐Windows13 support13 Therefore13 we13 suggest13 that13 you13 use13 a13 VNC13 client13 to13 access13 the13 virtual13 server13

Installing13 a13 VNC13 client13 should13 be13 done13 as13 the13 root13 user13 Use13 the13 yum13 command13 as13 follows13

[ibmcloudelkforcase ~]$ yum groupinstall GNOME Desktop

[ibmcloudelkforcase ~]$ yum install tigervnc-server

These13 commands13 first13 install13 a13 desktop13 style13 OS13 user13 interface13 and13 then13 a13 VNC13 client13 to13 remotely13 access13 the13 desktop13 style13 OS13

Then13 as13 either13 the13 root13 user13 or13 another13 user13 (example13 below13 uses13 elk)13 run13 the13 following13 command13 to13 create13 the13 VNC13 client13 connecon13

[elkelkforcase ~]$ vncserver

New elkforcase2 (elk) desktop is elkforcase2

Starting applications specified in homeelkvncxstartup

Log file is homeelkvncelkforcase2log

[elkelkforcase ~]$

Install13 a13 VNC13 client13 on13 the13 client13 machine13 used13 to13 access13 the13 Internet13 and13 (in13 the13 example13 above)13 use13 the13 command13 ltpublicIPadddressgt213 and13 the13 password13 for13 ELK13 to13 access13 the13 virtual13 server13

Using13 a13 VNC13 client13 means13 that13 you13 can13 leave13 long13 duraon13 installaons13 running13 without13 fear13 of13 the13 terminal13 losing13 connecon13 and13 the13 installaons13 failing13

Install13 the13 ELK13 stack13 Download13 the13 source13 soaware13 code13 for13 the13 ELK13 stack13 from13 the13 ELK13 website13 13

At13 the13 me13 of13 wring13 the13 downloaded13 soaware13 files13 were13

Page of 13 71 Page of 13 71

13

Transfer13 the13 files13 to13 the13 virtual13 server13 (using13 the13 scp13 method13 detailed13 above)13

Install13 Elas0csearch13 As13 a13 root13 user13 create13 a13 non-shy‐root13 user13 with13 the13 adduser13 command13

Create13 a13 directory13 to13 install13 Elascsearch13 into13 and13 untar13 the13 contents13 of13 the13 Elascsearch13 install13 file13 into13 that13 directory13

[elkelkforcase]$ cd home

[elkelkforcase]$ mkdir elasticsearch

[elkelkforcase]$ cp homeibmcloudelasticsearch-233targz homeelasticsearch

[elkelkforcase]$ cd elasticsearch

[elkelkforcase]$ gunzip elasticsearch-233targz

[elkelkforcase]$ untar elasticsearch-233tar

Elascsearch13 is13 now13 installed13

Start13 Elas0csearch13 To13 start13 Elascsearch13 go13 to13 the13 bin13 directory13 as13 the13 non-shy‐root13 user13 when13 you13 installed13 it13 and13 run13 the13 elascsearch13 executable13

[elkelkforcase]$ cd homeelasticsearchelasticsearch-233bin

[elkelkforcase]$ elasticsearch amp

[2] 22211

[elkelkforcase]$ [2016-07-05 133857730][INFO ][node] [Arsenal] version[233] pid[22211] build[218bdf12016-05-17T154004Z]

[2016-07-05 133857740][INFO ][node] [Arsenal] initializing

[2016-07-05 133858984][INFO ][plugins] [Arsenal] modules [reindex lang-expression lang-groovy] plugins [] sites []

[2016-07-05 133859146][INFO ][env] [Arsenal] using [1] data paths mounts [[ (rootfs)]] net usable_space [155gb] net total_space [199gb] spins [unknown] types [rootfs]

[2016-07-05 133859146][INFO ][env] [Arsenal] heap size [10156mb] compressed ordinary object pointers [true]

Page of 14 71 Page of 14 71

[2016-07-05 133859147][WARN ][env] [Arsenal] max file descriptors [4096] for elasticsearch process likely too low consider increasing to at least [65536]

[2016-07-05 133903526][INFO ][node] [Arsenal] initialized

[2016-07-05 133903526][INFO ][node] [Arsenal] starting

To13 test13 that13 Elascsearch13 is13 running13 correctly13 open13 the13 browser13 in13 the13 VNC13 client13 and13 run13 localhost920013

13

Install13 Kibana13 ELK13 recommends13 installing13 and13 running13 Kibana13 as13 root13 13

1 As13 root13 user13 create13 an13 installaon13 directory13 and13 untar13 the13 installaon13 file13

[rootelkforcase]$ cd home

[rootelkforcase]$ mkdir kibana

[rootelkforcase]$ cp homeibmcloudkibana-451-linux-x64targz homekibana

[rootelkforcase]$ cd kibana

[rootelkforcase]$ gunzip kibana-451-linux-x64targz

[rootelkforcase]$ untar kibana-451-linux-x64tar

2 Modify13 the13 manifestyml13 configuraon13 file13 Go13 to13 the13 kibana-shy‐451-shy‐linux-shy‐x6413 directory13

[rootelkforcase]$ cd homekibanakibana-451-linux-x64

3 Edit13 the13 manifestyml13 file13 so13 the13 contents13 are13

applicaons13

-shy‐ name13 kibana-shy‐in-shy‐bluemix13

memory13 128M13

host13 kibana-shy‐in-shy‐bluemix13

buildpack13 hcpsgithubcomcloudfoundry-shy‐communitynginx-shy‐buildpackgit13

Page of 15 71 Page of 15 71

4 Edit13 the13 configkibanayml13 file13 so13 the13 contents13 are13

Kibana is served by a back end server This controls which port to use

port 5601

The host to bind the server to

host 0000

If you are running kibana behind a proxy and want to mount it at a path

specify that path here The basePath cant end in a slash

serverbasePath

The maximum payload size in bytes on incoming server requests

servermaxPayloadBytes 1048576

The Elasticsearch instance to use for all your queries

elasticsearchurl httplocalhost9200

preserve_elasticsearch_host true will send the hostname specified in `elasticsearch` If you set it to false

then the host you use to connect to this Kibana instance will be sent

elasticsearchpreserveHost true

Kibana uses an index in Elasticsearch to store saved searches visualizations

and dashboards It will create a new index if it doesnt already exist

kibanaindex kibana

The default application to load

kibanadefaultAppId discover

If your Elasticsearch is protected with basic auth these are the user credentials

used by the Kibana server to perform maintenance on the kibana_index at startup Your

Kibana users will still need to authenticate with Elasticsearch (which is proxied through

the Kibana server)

Page of 16 71 Page of 16 71

elasticsearchusername user

elasticsearchpassword pass

SSL for outgoing requests from the Kibana Server to the browser (PEM formatted)

serversslcert pathtoyourservercrt

serversslkey pathtoyourserverkey

Optional setting to validate that your Elasticsearch backend uses the same key files (PEM formatted)

elasticsearchsslcert pathtoyourclientcrt

elasticsearchsslkey pathtoyourclientkey

If you need to provide a CA certificate for your Elasticsearch instance put

the path of the pem file here

elasticsearchsslca pathtoyourCApem

Set to false to have a complete disregard for the validity of the SSL

certificate

elasticsearchsslverify true

Time in milliseconds to wait for elasticsearch to respond to pings defaults to

request_timeout setting

elasticsearchpingTimeout 1500

Time in milliseconds to wait for responses from the back end or elasticsearch

This must be gt 0

elasticsearchrequestTimeout 30000

Time in milliseconds for Elasticsearch to wait for responses from shards

Set to 0 to disable

elasticsearchshardTimeout 0

Time in milliseconds to wait for Elasticsearch at Kibana startup before retrying

elasticsearchstartupTimeout 5000

elasticsearchstartupTimeout 30000

Page of 17 71 Page of 17 71

Set the path to where you would like the process id file to be created

pidfile varrunkibanapid

If you would like to send the log output to a file you can set the path below

loggingdest stdout

Set this to true to suppress all logging output

loggingsilent false

Set this to true to suppress all logging output except for error messages

loggingquiet false

Set this to true to log all events including system usage information and all requests

loggingverbose false

Kibana13 is13 now13 installed13

Start13 Kibana13 To13 start13 Kibana13 go13 to13 the13 bin13 directory13 and13 run13 the13 kibana13 executable13

[rootelkforcase]$

[rootelkforcase]$ cd homekibanakibana-451-linux-x64bin

[rootelkforcase]$ kibana amp

[1] 22283

[rootelkforcase] log [095900542] [info][status][pluginkibana] Status changed from uninitialized to green - Ready

log [095900600] [info][status][pluginelasticsearch] Status changed from uninitialized to yellow - Waiting for Elasticsearch

log [095900689] [info][status][pluginkbn_vislib_vis_types] Status changed from uninitialized to green - Ready

log [095900717] [info][status][pluginmarkdown_vis] Status changed from uninitialized to green - Ready

log [095900721] [info][status][pluginmetric_vis] Status changed from uninitialized to green - Ready

log [095900814] [info][status][pluginspyModes] Status changed from uninitialized to green - Ready

log [095900934] [info][status][pluginstatusPage] Status changed from uninitialized to green - Ready

Page of 18 71 Page of 18 71

log [095900941] [info][status][plugintable_vis] Status changed from uninitialized to green - Ready

log [095901067] [info][listening] Server running at http00005601

log [095906083] [info][status][pluginelasticsearch] Status changed from yellow to yellow - No existing Kibana index found

log [095908942] [info][status][pluginelasticsearch] Status changed from yellow to green - Kibana index ready

Kibana13 is13 now13 started13

Broadcast13 Bluemix13 app13 log13 records13 Bluemix13 apps13 (which13 are13 shown13 in13 the13 US13 South13 Bluemix13 region13 of13 the13 System13 Architecture13 diagram)13 will13 produce13 logs13 in13 a13 syslog13 rfc542413 format13 using13 a13 Bluemix13 tool13 called13 Loggregator13

For13 Logstash13 to13 receive13 these13 logs13 each13 Bluemix13 app13 needs13 to13 broadcast13 these13 logs13 To13 do13 that13 run13 the13 following13 commands13 on13 each13 Bluemix13 app13 using13 the13 public13 IP13 address13 of13 the13 ELK13 virtual13 server13 the13 ELK13 tag13 to13 trace13 the13 process13 and13 insert13 the13 name13 of13 each13 app13 in13 turn13

cf cups elk -l syslogltpublicIPaddress5000 cf bind-service ltmyappgt elk

cf restage ltmyappgt

Install13 Logstash13 To13 install13 Logstash13 as13 root13 user13 create13 a13 directory13 for13 the13 installaon13 and13 untar13 the13 installaon13 file13 there13

13 13 [rootelkforcase]$ cd home

[rootelkforcase]$ mkdir logstash

[rootelkforcase]$ cp homeibmcloudlogstash-233targz homelogstash

[rootelkforcase]$ cd logstash

[rootelkforcase]$ gunzip logstash-233targz

[rootelkforcase]$ untar logstash-233tar

Write13 Logstash13 configura0on13 Logstash13 requires13 a13 configuraon13 file13 to13 be13 wricen13 to13 receive13 transform13 and13 output13 the13 log13 records13 to13 Elascsearch13 This13 secon13 explains13 the13 basic13 requirements13 of13 processing13 Bluemix13 app13 log13 records13 into13 Elascsearch13 (more13 complex13 configuraons13 are13 detailed13 later13 in13 this13 document13 for13 the13 integraons13 with13 third-shy‐party13 tools)13

Create13 a13 conf13 file13 as13 root13 user13

[rootelkforcase]$ cd homelogstashlogstash-233

[rootelkforcase]$ mkdir config

Page of 19 71 Page of 19 71

[rootelkforcase]$ cd config

[rootelkforcase]$ touch loggregatorconf

Edit13 the13 loggregator13 file13 with13 the13 following13 contents13

13 Input13 secon13 defines13 which13 log13 records13 to13 receive13

13 port13 and13 type13 reflect13 method13 selected13 for13 Cloud13 Foundry13 to13 broadcast13 the13 records13

input

tcp

port =gt 5000

type =gt syslog

13 Filter13 secon13 ndash13 transforms13 the13 data13

filter

Add a field to identify the records as Cloud Foundry based

mutate

add_field =gt format =gt cf

13 Output13 secon13 ndash13 sends13 the13 data13 to13 its13 desnaon(s)13

output

13 Use13 the13 elascsearch13 plugin13 to13 send13 the13 data13 to13 the13 elascsearch13 instance13

13 elasticsearch

hosts =gt localhost

stdout codec =gt rubydebug

13 End13 of13 Config13

Save13 the13 file13 Verify13 the13 format13 of13 the13 file13 is13 acceptable13 by13 running13 the13 following13 commands13

[rootelkforcase]$ cd homelogstashlogstash-233

[rootelkforcase]$ binlogstash agent --configtest -f configloggregatorconf

Page of 20 71 Page of 20 71

Configuraon13 is13 OK13

Start13 Logstash13 Aaer13 you13 have13 wricen13 and13 verified13 the13 Logstash13 configuraon13 you13 can13 start13 Logstash13 to13 process13 records13 from13 the13 Bluemix13 apps13 (via13 Cloud13 Foundry)13 Run13 the13 following13 as13 root13 user13

[rootelkforcase]$ cd homelogstashlogstash-233

[rootelkforcase]$ binlogstash agent -f configloggregatorconf

Logstash13 is13 now13 started13

Monitor13 Logstash13 processing13 Once13 Logstash13 is13 running13 and13 log13 records13 are13 being13 produced13 by13 the13 Bluemix13 apps13 you13 can13 monitor13 progress13 in13 the13 same13 window13 that13 was13 used13 to13 start13 Logstash13 Output13 is13 similar13 to13 the13 following13

13

The13 log13 records13 being13 broadcast13 from13 the13 Bluemix13 apps13 are13 now13 sent13 to13 Elascsearch13 from13 the13 Logstash13 output13 configuraon13 Records13 are13 sent13 in13 real13 me13 If13 Logstash13 is13 not13 running13 while13 log13 records13 are13 being13 produced13 then13 these13 records13 will13 not13 be13 processed13 by13 Logstash13 and13 will13 not13 be13 stored13 in13 Elascsearch13

Page of 21 71 Page of 21 71

Use13 ELK13 The13 log13 records13 from13 the13 Bluemix13 apps13 are13 now13 being13 stored13 and13 indexed13 by13 the13 Elascsearch13 component13 This13 data13 can13 be13 accessed13 via13 the13 Kibana13 applicaon13

Kibana13 Discover13 page13 Using13 a13 web13 browser13 either13 directly13 from13 a13 client13 or13 using13 the13 VNC13 client13 access13 Kibana13 using13 the13 publicIPaddress13 of13 the13 ELK13 virtual13 server13

hcpltpublicIPaddressgt560113

No13 userpassword13 credenals13 are13 required13

Kibana13 defaults13 to13 a13 search13 of13 all13 records13 ()13 in13 the13 last13 1513 minutes13 and13 returns13 the13 results13

13

More13 complex13 searches13 are13 possible13 by13 changing13 the13 me13 filter13 and13 supplying13 search13 queries13

Change13 the13 0me13 filter13 To13 change13 the13 me13 ranges13 click13 the13 clock13 symbol13 in13 the13 top-shy‐right13 of13 the13 Kibana13 Discover13 page13

13

You13 will13 see13 the13 following13 opons13

13

Page of 22 71 Page of 22 71

A13 variety13 of13 quick13 me13 ranges13 can13 be13 selected13 or13 you13 can13 switch13 to13 relave13 or13 absolute13 me13 ranges13

Quick13 me13 ranges13 allow13 switching13 between13 last13 X13 minutes13 to13 last13 Y13 hours13 or13 Z13 days13 etc13

Relave13 me13 ranges13 allow13 more13 specific13 versions13 of13 the13 Quick13 opons13 from13 specific13 minuteshoursdays13 to13 now13

13

Absolute13 me13 ranges13 specify13 the13 start13 and13 end13 mes13 for13 the13 searches13

13

Custom13 search13 queries13 You13 can13 use13 the13 text13 window13 of13 the13 Discover13 page13 to13 search13 for13 matches13 within13 the13 log13 records13 You13 can13 use13 Apache13 Lucene13 or13 Elascsearch13 query13 DSL13 syntax13

13

Page of 23 71 Page of 23 71

Saved13 searches13 You13 can13 save13 any13 custom13 search13 syntax13 and13 choice13 of13 me13 range13 for13 future13 ease13 of13 use13 Any13 relave13 or13 absolute13 me13 frame13 is13 saved13 as13 part13 of13 the13 search13 13

To13 save13 a13 search13

1 Set13 up13 and13 run13 the13 required13 syntax13 and13 me13 frame13

13

2 Click13 the13 Save13 Search13 icon13

13

3 Type13 in13 a13 name13 for13 the13 saved13 search13

Page of 24 71 Page of 24 71

13

7 To13 delete13 or13 update13 saved13 searches13 click13 13 manage13 saved13 searches13

13

Use13 the13 edit13 icons13 to13 make13 any13 changes13

Page of 26 71 Page of 26 71

13

Build13 simple13 Kibana13 histograms13 Kibana13 allows13 you13 to13 build13 charts13 and13 dashboards13 based13 on13 the13 search13 capability13 offered13 by13 the13 Discover13 page13 Saved13 searches13 can13 be13 used13 as13 the13 basis13 for13 charts13 and13 dashboards13 built13 using13 those13 charts13 To13 build13 a13 simple13 histogram13

1 Select13 the13 Visualize13 page13

13

2 Select13 Ver0cal13 Bar13 Chart13

13

3 To13 use13 a13 saved13 search13 click13 From13 a13 saved13 search13

Page of 27 71 Page of 27 71

13

4 Select13 a13 previously13 saved13 search13

13

5 Select13 to13 edit13 the13 x-shy‐axis

6 Edit13 the13 fields13 with13 informaon13 about13 the13 aggregaon13 field13 interval13 and13 custom13 label13

Page of 28 71 Page of 28 71

13

Page of 29 71 Page of 29 71

7 Click13 the13 green13 arrow13 The13 histogram13 is13 created13 and13 run13 based13 on13 the13 saved13 search

13

8 To13 save13 the13 histogram13 click13 the13 Save13 Visualiza0on13 icon13 and13 add13 a13 tle13

13

13

9 Click13 Save13

10 To13 view13 the13 visualisaon13 click13 the13 Dashboard13 page13

Page of 30 71 Page of 30 71

13

11 Click13 the13 Add13 Visualiza0on13 icon13

13

12 Select13 the13 visiualizaon13 that13 you13 just13 created13

13

Page of 31 71 Page of 31 71

13

13 To13 save13 the13 dashboard13 click13 the13 Save13 Dashboard13 icon13

Page of 32 71 Page of 32 71

13

14 Add13 a13 tle13 and13 select13 whether13 to13 store13 the13 selected13 me13 frame13 with13 the13 dashboard13

13

15 Click13 Save13

16 You13 can13 edit13 the13 shape13 and13 number13 of13 charts13 and13 save13 the13 dashboard13

17 To13 re-shy‐run13 the13 dashboard13 select13 the13 Dashboard13 page13 and13 click13 the13 Load13 Saved13 Dashboard13 icon13

Page of 33 71 Page of 33 71

13

18 Select13 the13 saved13 dashboard13 from13 the13 list13

13

19 The13 dashboard13 runs13

13

Page of 34 71 Page of 34 71

Page of 35 71 Page of 35 71

Aler0ng13 with13 ELK13 ELK13 examines13 the13 log13 records13 in13 process13 to13 send13 alerts13 to13 third-shy‐party13 alerng13 event13 management13 and13 social13 media13 applicaons13 As13 shown13 previously13 in13 the13 ELK13 architecture13 the13 ELK13 strategy13 for13 alerng13 is13 to13 send13 alerts13 from13 the13 Logstash13 component13 directly13 Therefore13 log13 records13 and13 alerts13 are13 sent13 to13 both13 Elascsearch13 and13 third-shy‐party13 tools13 at13 the13 same13 instant13 using13 the13 output13 secon13 of13 the13 Logstash13 configuraon13 Any13 alerts13 are13 generated13 during13 the13 filter13 secon13 processing13 of13 the13 Logstash13 configuraon13

Create13 alerts13 in13 Logstash13 The13 Logstash13 filter13 secon13 is13 used13 to13 trigger13 alerts13 based13 on13 the13 log13 records13 being13 processed13 For13 example13 log13 records13 can13 be13 checked13 for13 errors13 warnings13 crashed13 or13 stopped13 messages13 and13 generate13 an13 alert13

Here13 is13 an13 example13 Logstash13 filter13 code13

13 if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

The13 specific13 tags13 are13 set13 against13 the13 log13 records13 to13 trigger13 the13 alert13 later13 in13 the13 configuraon13 in13 the13 output13 secon13

Control13 the13 number13 of13 generated13 alerts13 using13 the13 Logstash13 throcle13 plugin13 so13 that13 either13 a13 lot13 of13 instances13 in13 a13 fixed13 me13 period13 are13 required13 before13 an13 alert13 is13 sent13 or13 a13 specific13 number13 of13 instances13 triggers13 the13 alert13

Page of 36 71 Page of 36 71

Edit13 the13 plugin13 parameters13 to13 meet13 system13 requirements13

13 13 13 13 13 if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 10

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle13

after_count =gt 20

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 30

period =gt 1200

key =gt servicename_alarm_warning

Page of 37 71 Page of 37 71

add_tag =gt servicename_throttled

13 13 13 13 13 13 13 13 13

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 40

period =gt 1500

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

Enrich13 alerts13 with13 message13 detail13 Some13 of13 the13 third-shy‐party13 integraons13 benefit13 from13 having13 components13 of13 the13 raw13 log13 record13 sent13 in13 addion13 to13 the13 raw13 record13 or13 fields13 in13 the13 record13 edited13 transformed13 or13 translated13

Grok13 Logstash13 allows13 the13 log13 record13 to13 be13 broken13 up13 into13 logical13 secons13 and13 each13 secon13 stored13 in13 a13 new13 parameter13 using13 the13 grok13 parameter13 in13 the13 filter13 secon13 Here13 is13 an13 example13 grok13

13 grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

To13 build13 the13 grok13 statement13 Logstash13 offers13 an13 online13 tool13 at13 hcpsgrokdebugherokuappcom13

Translate13 If13 specific13 fields13 in13 the13 log13 record13 are13 coded13 (for13 example13 the13 Bluemix13 app13 name13 is13 oaen13 not13 sent13 in13 the13 syslog13 messages13 instead13 the13 GUID13 is13 sent13 which13 is13 not13 useful)13 then13 the13 filter13 secon13 translate13 plugin13 can13 be13 used13 to13 add13 new13 fields13 by13 holding13 the13 decoded13 versions13 of13 those13 values13

13 translate

field =gt data1

destination =gt data1

override =gt true

Page of 38 71 Page of 38 71

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

dc50f6e4-c052-436a-bf27-95ba392c4228 monverifyservicemanagement

0c677e82-5bfc-4e38-9155-e416249e3ee6 ms-catalog-postpyretic-pomiculture

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

However13 the13 translate13 plugin13 is13 not13 included13 with13 the13 Logstash13 installaon13 In13 the13 next13 secon13 I13 will13 13 describe13 how13 to13 add13 new13 plugins13 to13 the13 Logstash13 instance13

Add13 custom13 plugins13 to13 Logstash13 As13 an13 open13 source13 tool13 Logstash13 encourages13 users13 to13 develop13 content13 for13 all13 users13 to13 share13 Some13 of13 the13 plugins13 that13 were13 created13 by13 our13 community13 include13 a13 translaon13 plugin13 an13 output13 Slack13 plugin13 and13 an13 output13 syslog13 plugin13 Letrsquos13 look13 at13 how13 to13 add13 these13 plugins13 to13 your13 environment13

Filter13 translate13 plugin13 To13 add13 the13 translate13 plugin13

1 Download13 the13 plugin13

2 Copy13 the13 plugin13 file13 logstash-shy‐filter-shy‐translate-shy‐214gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-filter-translate 214

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Output13 Slack13 plugin13 To13 add13 the13 Slack13 plugin13

Page of 39 71 Page of 39 71

1 Download13 the13 plugin13 13

2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐slack-shy‐014gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-output-slack 014

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Output13 syslog13 plugin13 To13 add13 the13 syslog13 plugin13

1 Download13 the13 plugin13 13

2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐syslog-shy‐214gem13 to13 the13 ELK13 Virtual13 Server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-output-syslog 214

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Integrate13 Logstash13 with13 Slack13 To13 send13 generated13 alerts13 to13 Slack13 complete13 these13 steps13

1 Create13 a13 Slack13 integraon13 13

2 Use13 the13 Slack13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 Slack13 integraon13

113 Login13 to13 Slack13 and13 go13 to13 the13 Apps13 page13 13

Page of 40 71 Page of 40 71

13

313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13

13

A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13

13

Page of 42 71 Page of 42 71

The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13

13

The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13

13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

slack

url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5

channel =gt cloud-operations-mon

username =gt nick_cawood

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13

Page of 43 71 Page of 43 71

Page of 44 71 Page of 44 71

In13 Slack13 the13 alerts13 look13 like13 this13

13

Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13

1 Create13 a13 PagerDuty13 integraon13 13

2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13

1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13

13

2 Click13 Add13 New13 Service13

13

313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13

Page of 45 71 Page of 45 71

13

413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13

513 Click13 Add13 Service13

13

The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13

Page of 46 71 Page of 46 71

13

The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt be7eb1ea9c224018923e818d3e891d0b

incident_key =gt logstashhosttype

Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13

Page of 47 71 Page of 47 71

13

Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13

1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13

2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13

3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13

4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13

13 13 http

url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo

http_method =gt post

format =gt json

For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13

13 13 syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

Page of 48 71 Page of 48 71

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13

13

The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13

if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2

This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13

Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13

Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13

Logstash13 severity Shown13 in13 Omnibus13 as

alert Indeterminate

crical Warning

debug 7

emergency Clear

error Minor

informaonal 6

noce Crical

warning Major

Page of 49 71 Page of 49 71

13

Page of 50 71 Page of 50 71

Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13

input

tcp

port =gt 5000

type =gt syslog

filter

mutate

add_field =gt format =gt cf

grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

translate

field =gt data1

destination =gt data1

override =gt true

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

=== Check Alarm Criteria ===

Standard Key Words

if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

=== Alarm Exclusions ===

if [message] =~ (i)sometexthere

mutate remove_tag =gt servicename_alarm

Throttle alarms so we do not hit PD rate limits

Page of 52 71 Page of 52 71

if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 5

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle

after_count =gt 5

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 5

period =gt 1200

key =gt servicename_alarm_warning

add_tag =gt servicename_throttled

Page of 53 71 Page of 53 71

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

output

elasticsearch

hosts =gt localhost

stdout codec =gt rubydebug

if servicename_alarm_err in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

Page of 54 71 Page of 54 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 55 71 Page of 55 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

Page of 56 71 Page of 56 71

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert err message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_error in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

Page of 57 71 Page of 57 71

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 58 71 Page of 58 71

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 59 71 Page of 59 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert error message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

Page of 60 71 Page of 60 71

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

Page of 61 71 Page of 61 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 62 71 Page of 62 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt x

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert fatal message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

Page of 63 71 Page of 63 71

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

Page of 64 71 Page of 64 71

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 65 71 Page of 65 71

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt heavy_exclamation_mark

Page of 66 71 Page of 66 71

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert WARNING message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 67 71 Page of 67 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

Page of 68 71 Page of 68 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 69 71 Page of 69 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

Page of 70 71 Page of 70 71

Page of 71 71 Page of 71 71

Page 2: How to integrate the Elasticsearch Logstash Kibana (ELK ... · How to integrate the Elasticsearch Logstash Kibana (ELK) log analytics stack into IBM Bluemix Nick Cawood IBM Cloud

CONTENTS13

Introducon13 313

What13 is13 Elascsearch13 Logstash13 Kibana13 313

LOGSTASH 3 ELASTICSEARCH 4 KIBANA 4 ELK SYSTEM ARCHITECTURE 4

Service13 Management13 architecture13 613

Log13 Monitoring13 System13 architecture13 613

Bluemix13 environment13 system13 architecture13 613

Creang13 a13 virtual13 server13 in13 Bluemix13 713

CREATE AN SSH KEY 8 PROVISIONING A VIRTUAL SERVER 9 SET UP THE VIRTUAL SERVER 12

Log on to the virtual server 12 Acces the root user 12 Copy files to the virtual server 13 Install and access the virtual server via VNC server 13

Install13 the13 ELK13 stack13 1313

INSTALL ELASTICSEARCH 14 Start Elasticsearch 14

INSTALL KIBANA 15 Start Kibana 18

BROADCAST BLUEMIX APP LOG RECORDS 19 INSTALL LOGSTASH 19

Write Logstash configuration 19 Start Logstash 21 Monitor Logstash processing 21

Use13 ELK13 2213

KIBANA DISCOVER PAGE 22 Change the time filter 22 Custom search queries 23 Saved searches 24

BUILD SIMPLE KIBANA HISTOGRAMS 27 Alerng13 with13 ELK13 3613

CREATE ALERTS IN LOGSTASH 36 ENRICH ALERTS WITH MESSAGE DETAIL 38

Grok 38 Translate 38

ADD CUSTOM PLUGINS TO LOGSTASH 39 Filter translate plugin 39 Output Slack plugin 39 Output syslog plugin 40

INTEGRATE LOGSTASH WITH SLACK 40 INTEGRATE LOGSTASH WITH PAGERDUTY 45 INTEGRATE LOGSTASH WITH IBM NETCOOL OMNIBUS 48

Appendix13 Logstash13 configuraon13 51

Page of 2 71

Introduc0on13 IBMreg13 Bluemixreg13 IBMrsquos13 PlaRorm13 as13 a13 Service13 (PaaS)13 allows13 you13 to13 rapidly13 build13 and13 deploy13 an13 applicaon13 Applicaons13 can13 run13 in13 a13 nave13 or13 a13 hybrid13 Bluemix13 environment13 13 13

This13 document13 describes13 the13 integraon13 of13 the13 Elascsearch13 Logstash13 Kibana13 (ELK)13 log13 analycs13 open13 source13 stack13 into13 the13 Bluemix13 environment13 how13 to13 collect13 logs13 from13 Bluemix13 applicaons13 (apps)13 and13 send13 alerts13 from13 those13 log13 records13 to13 selected13 alertevent13 noficaon13 targets13 (PagerDuty13 Slack13 and13 IBM13 Netcool13 Omnibus)13

This13 document13 describes13 how13 to13 provision13 Bluemix13 virtual13 servers13 in13 the13 Bluemix13 environment13 install13 the13 ELK13 stack13 on13 a13 Bluemix13 virtual13 server13 and13 configure13 the13 ELK13 stack13 to13 integrate13 with13 other13 components13 deployed13 on13 the13 Bluemix13 environment13

The13 steps13 to13 integrate13 the13 ELK13 stack13 into13 the13 Cloud13 Architecture13 and13 Soluon13 Engineering13 (CASE)13 environment13 are13 used13 as13 the13 basis13 for13 this13 document13 13 For13 reference13 both13 the13 ELK13 architecture13 within13 the13 CASE13 environment13 and13 the13 whole13 CASE13 environment13 are13 defined13 using13 system13 architecture13 diagrams13 as13 well13 as13 diagrams13 defining13 the13 ELK13 structure13 and13 ELK13 integraon13 into13 Bluemix13 architecture13

What13 is13 Elas0csearch13 Logstash13 Kibana13 Elascsearch13 Logstash13 Kibana13 is13 an13 open13 source13 stack13 of13 three13 applicaons13 that13 work13 together13 to13 provide13 an13 end-shy‐to-shy‐end13 search13 and13 visualisaon13 plaRorm13 for13 the13 invesgaon13 of13 log13 file13 sources13 in13 real13 me13 Log13 file13 records13 can13 be13 searched13 from13 a13 variety13 of13 log13 sources13 charts13 and13 dashboards13 built13 to13 visualize13 the13 log13 records13 The13 three13 applicaons13 are13 Elascsearch13 Logstash13 and13 Kibana13

Logstash13 Logstash13 is13 an13 applicaon13 that13 allows13 the13 ELK13 stack13 to13 manage13 the13 gathering13 transformaon13 and13 transport13 of13 log13 file13 records13 Logstash13 is13 able13 to13 pull13 records13 from13 remote13 sources13 or13 listen13 for13 records13 on13 specified13 ports13

Logstash13 has13 three13 areas13 of13 funcon13

bull Process13 input13 of13 the13 log13 file13 records13 from13 the13 remote13 source13 You13 can13 configure13 Logstash13 to13 receive13 records13 in13 a13 variety13 of13 methods13 for13 example13 reading13 from13 a13 flat13 file13 or13 from13 a13 TCP13 port13 or13 direct13 from13 a13 remote13 or13 local13 syslog13 These13 methods13 are13 defined13 in13 the13 Logstash13 online13 support13 and13 input13 secon13 of13 the13 Logstash13 conf13 file13 Once13 all13 input13 processing13 is13 completed13 Logstash13 moves13 onto13 the13 next13 funcon13

bull Transform13 and13 enrich13 the13 log13 records13 that13 take13 place13 in13 the13 Logstash13 filter13 secon13 Any13 number13 of13 changes13 to13 the13 format13 (and13 content13 if13 required)13 of13 the13 log13 file13 record13 can13 be13 made13 using13 the13 various13 methods13 provided13 with13 Logstash13 for13 example13 methods13 to13 add13 and13 enrich13 log13 record13 fields13 and13 the13 flagging13 I13 describe13 excluding13 and13 liming13 records13 for13 alerng13 later13 in13 this13 document13 Aaer13 the13 log13 record13 is13 transformed13 Logstash13 processes13 the13 record13

bull Send13 the13 log13 records13 to13 the13 Elascsearch13 applicaon13 from13 the13 output13 secon13 of13 the13 Logstash13 configuraon13 A13 specific13 elascsearch13 method13 is13 included13 with13 the13 Logstash13 instance13 that13 sends13 the13 log13 records13 to13 the13 Elascsearch13 instance13 However13 as13 with13 the13 input13 and13 filter13

Page of 3 71 Page of 3 71

secons13 a13 variety13 of13 methods13 are13 available13 to13 process13 the13 data13 sent13 from13 the13 filter13 secon13 For13 example13 the13 records13 can13 be13 sent13 to13 external13 alerng13 tools13 such13 as13 PagerDuty13 social13 media13 tools13 such13 as13 Slack13 or13 event13 management13 hubs13 like13 IBM13 Netcool13 Omnibus13 via13 output13 methods13 syslog13 or13 hcp13 13

As13 an13 open13 source13 tool13 non-shy‐Logstash13 employees13 develop13 other13 methods13 and13 share13 them13 with13 the13 user13 community13 for13 addion13 to13 the13 library13 of13 methods13 Logstash13 can13 execute13 (examples13 of13 this13 process13 are13 included13 later13 in13 this13 document)13

Elas0csearch13 Elascsearch13 is13 the13 index13 store13 and13 query13 applicaon13 on13 the13 ELK13 stack13 It13 provides13 a13 standard13 full13 text13 search13 facility13 of13 the13 log13 file13 records13 processed13 by13 Logstash13 The13 ELK13 online13 help13 documentaon13 describes13 Elascsearch13 as13 follows13 ldquoIt13 is13 generally13 used13 as13 the13 underlying13 enginetechnology13 that13 powers13 applicaons13 that13 have13 complex13 search13 features13 and13 requirementsrdquo13

Kibana13 Kibana13 is13 the13 user13 interface13 for13 the13 ELK13 stack13 Kibana13 integrates13 with13 Elascsearch13 to13 visualise13 the13 results13 of13 searches13 and13 allow13 the13 creaon13 of13 charts13 and13 dashboards13 based13 on13 the13 searches13 The13 searches13 charts13 and13 dashboard13 can13 visualise13 real-shy‐me13 or13 historical13 log13 records13 Users13 access13 Kibana13 through13 an13 hcp13 URL13 from13 a13 client13 web13 browser13

ELK13 System13 Architecture13 The13 ELK13 stack13 can13 be13 described13 as13 follows13

Page of 4 71 Page of 4 71

13

Page of 5 71 Page of 5 71

Service13 Management13 architecture13 The13 IBM13 Cloud13 Architecture13 Center13 created13 a13 standard13 reference13 architecture13 for13 how13 to13 set13 up13 your13 cloud13 environemnt13 to13 handle13 incident13 management13 13

View13 the13 standard13 Service13 Management13 architecure13

13 13

Log13 Monitoring13 System13 architecture13 The13 following13 diagram13 shows13 a13 standard13 architecture13 for13 the13 log-shy‐monitoring13 system13 described13 in13 this13 tutorial13 13

13

Bluemix13 environment13 system13 architecture13 The13 ELK13 stack13 can13 be13 installed13 into13 a13 variety13 of13 environments13 When13 using13 Bluemix13 the13 following13 figure13 shows13 what13 the13 environment13 might13 look13 like13

Page of 6 71 Page of 6 71

13

The13 Bluemix13 cloud13 plaRorm13 offers13 a13 Public13 network13 and13 access13 to13 US13 South13 and13 UK13 environments13 From13 all13 these13 environments13 third-shy‐party13 cloud-shy‐based13 soluons13 are13 also13 accessible13

The13 ELK13 stack13 is13 installed13 in13 the13 Bluemix13 United13 Kingdom13 environment13 because13 this13 is13 the13 environment13 where13 Bluemix13 has13 virtual13 servers13 available13 for13 provisioning13 Other13 applicaons13 in13 the13 CASE13 project13 were13 also13 installed13 on13 virtual13 servers13 in13 the13 Bluemix13 United13 Kingdom13 region13

Bluemix13 apps13 were13 used13 to13 create13 a13 system13 to13 be13 monitored13 by13 ELK13 These13 apps13 were13 hosted13 in13 the13 Bluemix13 US13 South13 environment13 using13 Bluemix13 microservices13 which13 produce13 logs13 during13 normal13 operaon13 and13 during13 crashes13 and13 outages13

Through13 the13 Bluemix13 Cloud13 Foundry13 process13 the13 log13 records13 can13 be13 broadcast13 from13 the13 Bluemix13 apps13 so13 Logstash13 can13 receive13 them13 and13 send13 them13 to13 Elascsearch13 within13 the13 ELK13 stack13 or13 other13 virtual13 servers13 (hosng13 other13 IBM13 or13 third-shy‐party13 products)13 in13 the13 Bluemix13 United13 Kingdom13 environment13 or13 to13 third-shy‐party13 applicaons13 in13 the13 public13 cloud13 (Pager13 Duty13 Slack13 etc)13

Crea0ng13 a13 virtual13 server13 in13 Bluemix13 With13 Bluemix13 you13 can13 provision13 a13 virtual13 server13 in13 the13 cloud13 to13 install13 standalone13 or13 integrated13 system13 components13 Not13 all13 Bluemix13 regions13 offer13 this13 service13 all13 the13 me13 due13 to13 beta-shy‐project13 constraints13 In13 this13 arclersquos13 examples13 virtual13 servers13 were13 allowed13 to13 be13 provisioned13 in13 the13 United13 Kingdom13 region13 For13 more13 informaon13 read13 how13 to13 provision13 a13 Bluemix13 virtual13 server13 including13 some13 simple13 setup13 steps13 for13 ease13 of13 use13

Page of 7 71 Page of 7 71

Create13 an13 SSH13 key13 To13 access13 your13 virtual13 server13 once13 it13 is13 provisioned13 allocate13 an13 SSH13 key13 to13 the13 virtual13 server13

To13 generate13 a13 publicprivate13 RSA13 key13 pair13 follow13 these13 steps13

1 In13 a13 UNIX13 or13 Linuxreg13 window13 run13 the13 following13 command13 specifying13 a13 ltuniquekeynamegt13 of13 your13 your13 choice13 and13 a13 password13 or13 passphrase13 (memorize13 the13 password)$13 ssh-shy‐keygen13 -shy‐t13 rsa13 -shy‐f13 ltuniquekeynamegt13

2 Enter13 the13 passphrase13 (empty13 for13 no13 passphrase)13

3 Enter13 the13 same13 passphrase13 again13

Your13 idenficaon13 has13 been13 saved13 in13 ltuniquekeynamegt13

Your13 public13 key13 has13 been13 saved13 in13 ltuniquekeynamegtpub13

The13 key13 fingerprint13 is13

d6268464506525930ce4632c693c203c13 ltusergtltservergt13

The13 keys13 randomart13 image13 is13

+--[ RSA 2048]----+

|o ++=o |

| E o ooo |

| |

| + o |

| S o |

| o |

| |

| |

| |

+-----------------+

Two13 keys13 are13 created13 public13 and13 private13 The13 public13 key13 contents13 are13 used13 to13 create13 a13 virtual13 server13 with13 a13 public13 IP13 address13 The13 private13 key13 is13 used13 to13 access13 the13 virtual13 server13 from13 a13 client13 machine13

Page of 8 71 Page of 8 71

Provisioning13 a13 virtual13 server13 1 A13 Bluemix13 account13 is13 required13 for13 this13 process13 Sign13 up13 here13 and13 log13 in13 Make13 sure13 your13

environment13 is13 correct13 For13 example13 if13 you13 are13 in13 the13 United13 Kingdom13 select13 the13 United13 Kingdom13 region13 in13 the13 Bluemix13 console13

13

13

2 Add13 a13 virtual13 server13 to13 your13 Bluemix13 environment13 Click13 Run13 Virtual13 Servers13

13

3 Select13 an13 OS13 image13 from13 the13 EXISTING13 list13 For13 ELK13 select13 Centos13 OS13 (use13 latest13 available13 version)13

Page of 9 71 Page of 9 71

4 Name13 the13 virtual13 server13

5 Make13 sure13 Assign13 public13 IP13 address13 is13 checked13

13

6 Click13 Add13 Key13 to13 add13 the13 SSH13 key13 you13 created13 previously13 Copy13 the13 contents13 of13 the13 public13 key13 into13 the13 Public13 Key13 to13 import13 field13

Page of 10 71 Page of 10 71

13

7 Click13 OK13 Then13 click13 CREATE13

The13 virtual13 server13 is13 provisioned13 and13 added13 to13 the13 Bluemix13 dashboard13 under13 VIRTUAL13 SERVERS13

Page of 11 71 Page of 11 71

13

Set13 up13 the13 virtual13 server13

Log13 on13 to13 the13 virtual13 server13 Now13 you13 can13 log13 on13 to13 the13 virtual13 server13 and13 add13 new13 users13 and13 programs13 to13 make13 it13 easier13 to13 access13 the13 virtual13 server13

To13 access13 the13 server13 go13 to13 the13 UNIX13 or13 Linux13 window13 that13 you13 used13 to13 create13 the13 SSH13 key13 In13 the13 directory13 where13 the13 ltuniquekeynamegt13 files13 exist13 run13 the13 following13 command13 referencing13 the13 SSH13 keyname13 the13 public13 IP13 address13 Bluemix13 allocated13 for13 the13 virtual13 server13 and13 the13 password13 for13 the13 SSH13 key13

[userserver ~]$ ssh -i ltuniquekeynamegt ibmcloudltpublicIPaddressgt

Enter13 the13 passphrase13 for13 the13 key13 ltuniquekeynamegt13

Last login Tue Jul 5 125546 2016 from 108-210-96-231lightspeedrlghncsbcglobalnet

[ibmcloudelkforcase ~]$

ibmcloud13 is13 the13 default13 user13 for13 the13 Bluemix13 virtual13 servers13 and13 allows13 SSH13 to13 access13 the13 server13

Acces13 the13 root13 user13 To13 access13 the13 root13 user13 log13 on13 as13 ibmcloud13 user13 and13 run13 the13 following13 command13

[ibmcloudelkforcase ~]$ sudo bash

[rootelkforcase ibmcloud]

The13 root13 user13 can13 be13 used13 to13 add13 programs13 to13 the13 server13 and13 create13 and13 edit13 users13 as13 required13 to13 access13 applicaons13 installed13 on13 the13 virtual13 server13

Page of 12 71 Page of 12 71

Copy13 files13 to13 the13 virtual13 server13 To13 move13 installaon13 files13 to13 the13 server13 go13 to13 the13 UNIX13 or13 Linux13 window13 you13 used13 to13 create13 the13 SSH13 key13 and13 the13 directory13 where13 the13 ltuniquekeynamegt13 files13 exist13 Move13 the13 installaon13 files13 to13 the13 same13 locaon13

Run13 the13 following13 command13 referencing13 the13 SSH13 keyname13 the13 public13 IP13 address13 Bluemix13 allocated13 for13 the13 virtual13 server13 and13 the13 password13 for13 the13 SSH13 key13

[userserver ~]$ scp ndashi ltuniquekeynamegt ltINSTALLATIONFILENAMEgt ibmcloudltpublicIPaddressgthomeibmcloud

Enter13 the13 passphrase13 for13 key13 cloudkey313

This13 copies13 the13 file13 using13 the13 ibmcloud13 user13 to13 the13 homeibmcloud13 directory13 on13 the13 virtual13 server13

Install13 and13 access13 the13 virtual13 server13 via13 VNC13 server13 Using13 the13 ssh13 command13 to13 access13 the13 virtual13 server13 is13 not13 the13 most13 stable13 method13 and13 does13 not13 offer13 X-shy‐Windows13 support13 Therefore13 we13 suggest13 that13 you13 use13 a13 VNC13 client13 to13 access13 the13 virtual13 server13

Installing13 a13 VNC13 client13 should13 be13 done13 as13 the13 root13 user13 Use13 the13 yum13 command13 as13 follows13

[ibmcloudelkforcase ~]$ yum groupinstall GNOME Desktop

[ibmcloudelkforcase ~]$ yum install tigervnc-server

These13 commands13 first13 install13 a13 desktop13 style13 OS13 user13 interface13 and13 then13 a13 VNC13 client13 to13 remotely13 access13 the13 desktop13 style13 OS13

Then13 as13 either13 the13 root13 user13 or13 another13 user13 (example13 below13 uses13 elk)13 run13 the13 following13 command13 to13 create13 the13 VNC13 client13 connecon13

[elkelkforcase ~]$ vncserver

New elkforcase2 (elk) desktop is elkforcase2

Starting applications specified in homeelkvncxstartup

Log file is homeelkvncelkforcase2log

[elkelkforcase ~]$

Install13 a13 VNC13 client13 on13 the13 client13 machine13 used13 to13 access13 the13 Internet13 and13 (in13 the13 example13 above)13 use13 the13 command13 ltpublicIPadddressgt213 and13 the13 password13 for13 ELK13 to13 access13 the13 virtual13 server13

Using13 a13 VNC13 client13 means13 that13 you13 can13 leave13 long13 duraon13 installaons13 running13 without13 fear13 of13 the13 terminal13 losing13 connecon13 and13 the13 installaons13 failing13

Install13 the13 ELK13 stack13 Download13 the13 source13 soaware13 code13 for13 the13 ELK13 stack13 from13 the13 ELK13 website13 13

At13 the13 me13 of13 wring13 the13 downloaded13 soaware13 files13 were13

Page of 13 71 Page of 13 71

13

Transfer13 the13 files13 to13 the13 virtual13 server13 (using13 the13 scp13 method13 detailed13 above)13

Install13 Elas0csearch13 As13 a13 root13 user13 create13 a13 non-shy‐root13 user13 with13 the13 adduser13 command13

Create13 a13 directory13 to13 install13 Elascsearch13 into13 and13 untar13 the13 contents13 of13 the13 Elascsearch13 install13 file13 into13 that13 directory13

[elkelkforcase]$ cd home

[elkelkforcase]$ mkdir elasticsearch

[elkelkforcase]$ cp homeibmcloudelasticsearch-233targz homeelasticsearch

[elkelkforcase]$ cd elasticsearch

[elkelkforcase]$ gunzip elasticsearch-233targz

[elkelkforcase]$ untar elasticsearch-233tar

Elascsearch13 is13 now13 installed13

Start13 Elas0csearch13 To13 start13 Elascsearch13 go13 to13 the13 bin13 directory13 as13 the13 non-shy‐root13 user13 when13 you13 installed13 it13 and13 run13 the13 elascsearch13 executable13

[elkelkforcase]$ cd homeelasticsearchelasticsearch-233bin

[elkelkforcase]$ elasticsearch amp

[2] 22211

[elkelkforcase]$ [2016-07-05 133857730][INFO ][node] [Arsenal] version[233] pid[22211] build[218bdf12016-05-17T154004Z]

[2016-07-05 133857740][INFO ][node] [Arsenal] initializing

[2016-07-05 133858984][INFO ][plugins] [Arsenal] modules [reindex lang-expression lang-groovy] plugins [] sites []

[2016-07-05 133859146][INFO ][env] [Arsenal] using [1] data paths mounts [[ (rootfs)]] net usable_space [155gb] net total_space [199gb] spins [unknown] types [rootfs]

[2016-07-05 133859146][INFO ][env] [Arsenal] heap size [10156mb] compressed ordinary object pointers [true]

Page of 14 71 Page of 14 71

[2016-07-05 133859147][WARN ][env] [Arsenal] max file descriptors [4096] for elasticsearch process likely too low consider increasing to at least [65536]

[2016-07-05 133903526][INFO ][node] [Arsenal] initialized

[2016-07-05 133903526][INFO ][node] [Arsenal] starting

To13 test13 that13 Elascsearch13 is13 running13 correctly13 open13 the13 browser13 in13 the13 VNC13 client13 and13 run13 localhost920013

13

Install13 Kibana13 ELK13 recommends13 installing13 and13 running13 Kibana13 as13 root13 13

1 As13 root13 user13 create13 an13 installaon13 directory13 and13 untar13 the13 installaon13 file13

[rootelkforcase]$ cd home

[rootelkforcase]$ mkdir kibana

[rootelkforcase]$ cp homeibmcloudkibana-451-linux-x64targz homekibana

[rootelkforcase]$ cd kibana

[rootelkforcase]$ gunzip kibana-451-linux-x64targz

[rootelkforcase]$ untar kibana-451-linux-x64tar

2 Modify13 the13 manifestyml13 configuraon13 file13 Go13 to13 the13 kibana-shy‐451-shy‐linux-shy‐x6413 directory13

[rootelkforcase]$ cd homekibanakibana-451-linux-x64

3 Edit13 the13 manifestyml13 file13 so13 the13 contents13 are13

applicaons13

-shy‐ name13 kibana-shy‐in-shy‐bluemix13

memory13 128M13

host13 kibana-shy‐in-shy‐bluemix13

buildpack13 hcpsgithubcomcloudfoundry-shy‐communitynginx-shy‐buildpackgit13

Page of 15 71 Page of 15 71

4 Edit13 the13 configkibanayml13 file13 so13 the13 contents13 are13

Kibana is served by a back end server This controls which port to use

port 5601

The host to bind the server to

host 0000

If you are running kibana behind a proxy and want to mount it at a path

specify that path here The basePath cant end in a slash

serverbasePath

The maximum payload size in bytes on incoming server requests

servermaxPayloadBytes 1048576

The Elasticsearch instance to use for all your queries

elasticsearchurl httplocalhost9200

preserve_elasticsearch_host true will send the hostname specified in `elasticsearch` If you set it to false

then the host you use to connect to this Kibana instance will be sent

elasticsearchpreserveHost true

Kibana uses an index in Elasticsearch to store saved searches visualizations

and dashboards It will create a new index if it doesnt already exist

kibanaindex kibana

The default application to load

kibanadefaultAppId discover

If your Elasticsearch is protected with basic auth these are the user credentials

used by the Kibana server to perform maintenance on the kibana_index at startup Your

Kibana users will still need to authenticate with Elasticsearch (which is proxied through

the Kibana server)

Page of 16 71 Page of 16 71

elasticsearchusername user

elasticsearchpassword pass

SSL for outgoing requests from the Kibana Server to the browser (PEM formatted)

serversslcert pathtoyourservercrt

serversslkey pathtoyourserverkey

Optional setting to validate that your Elasticsearch backend uses the same key files (PEM formatted)

elasticsearchsslcert pathtoyourclientcrt

elasticsearchsslkey pathtoyourclientkey

If you need to provide a CA certificate for your Elasticsearch instance put

the path of the pem file here

elasticsearchsslca pathtoyourCApem

Set to false to have a complete disregard for the validity of the SSL

certificate

elasticsearchsslverify true

Time in milliseconds to wait for elasticsearch to respond to pings defaults to

request_timeout setting

elasticsearchpingTimeout 1500

Time in milliseconds to wait for responses from the back end or elasticsearch

This must be gt 0

elasticsearchrequestTimeout 30000

Time in milliseconds for Elasticsearch to wait for responses from shards

Set to 0 to disable

elasticsearchshardTimeout 0

Time in milliseconds to wait for Elasticsearch at Kibana startup before retrying

elasticsearchstartupTimeout 5000

elasticsearchstartupTimeout 30000

Page of 17 71 Page of 17 71

Set the path to where you would like the process id file to be created

pidfile varrunkibanapid

If you would like to send the log output to a file you can set the path below

loggingdest stdout

Set this to true to suppress all logging output

loggingsilent false

Set this to true to suppress all logging output except for error messages

loggingquiet false

Set this to true to log all events including system usage information and all requests

loggingverbose false

Kibana13 is13 now13 installed13

Start13 Kibana13 To13 start13 Kibana13 go13 to13 the13 bin13 directory13 and13 run13 the13 kibana13 executable13

[rootelkforcase]$

[rootelkforcase]$ cd homekibanakibana-451-linux-x64bin

[rootelkforcase]$ kibana amp

[1] 22283

[rootelkforcase] log [095900542] [info][status][pluginkibana] Status changed from uninitialized to green - Ready

log [095900600] [info][status][pluginelasticsearch] Status changed from uninitialized to yellow - Waiting for Elasticsearch

log [095900689] [info][status][pluginkbn_vislib_vis_types] Status changed from uninitialized to green - Ready

log [095900717] [info][status][pluginmarkdown_vis] Status changed from uninitialized to green - Ready

log [095900721] [info][status][pluginmetric_vis] Status changed from uninitialized to green - Ready

log [095900814] [info][status][pluginspyModes] Status changed from uninitialized to green - Ready

log [095900934] [info][status][pluginstatusPage] Status changed from uninitialized to green - Ready

Page of 18 71 Page of 18 71

log [095900941] [info][status][plugintable_vis] Status changed from uninitialized to green - Ready

log [095901067] [info][listening] Server running at http00005601

log [095906083] [info][status][pluginelasticsearch] Status changed from yellow to yellow - No existing Kibana index found

log [095908942] [info][status][pluginelasticsearch] Status changed from yellow to green - Kibana index ready

Kibana13 is13 now13 started13

Broadcast13 Bluemix13 app13 log13 records13 Bluemix13 apps13 (which13 are13 shown13 in13 the13 US13 South13 Bluemix13 region13 of13 the13 System13 Architecture13 diagram)13 will13 produce13 logs13 in13 a13 syslog13 rfc542413 format13 using13 a13 Bluemix13 tool13 called13 Loggregator13

For13 Logstash13 to13 receive13 these13 logs13 each13 Bluemix13 app13 needs13 to13 broadcast13 these13 logs13 To13 do13 that13 run13 the13 following13 commands13 on13 each13 Bluemix13 app13 using13 the13 public13 IP13 address13 of13 the13 ELK13 virtual13 server13 the13 ELK13 tag13 to13 trace13 the13 process13 and13 insert13 the13 name13 of13 each13 app13 in13 turn13

cf cups elk -l syslogltpublicIPaddress5000 cf bind-service ltmyappgt elk

cf restage ltmyappgt

Install13 Logstash13 To13 install13 Logstash13 as13 root13 user13 create13 a13 directory13 for13 the13 installaon13 and13 untar13 the13 installaon13 file13 there13

13 13 [rootelkforcase]$ cd home

[rootelkforcase]$ mkdir logstash

[rootelkforcase]$ cp homeibmcloudlogstash-233targz homelogstash

[rootelkforcase]$ cd logstash

[rootelkforcase]$ gunzip logstash-233targz

[rootelkforcase]$ untar logstash-233tar

Write13 Logstash13 configura0on13 Logstash13 requires13 a13 configuraon13 file13 to13 be13 wricen13 to13 receive13 transform13 and13 output13 the13 log13 records13 to13 Elascsearch13 This13 secon13 explains13 the13 basic13 requirements13 of13 processing13 Bluemix13 app13 log13 records13 into13 Elascsearch13 (more13 complex13 configuraons13 are13 detailed13 later13 in13 this13 document13 for13 the13 integraons13 with13 third-shy‐party13 tools)13

Create13 a13 conf13 file13 as13 root13 user13

[rootelkforcase]$ cd homelogstashlogstash-233

[rootelkforcase]$ mkdir config

Page of 19 71 Page of 19 71

[rootelkforcase]$ cd config

[rootelkforcase]$ touch loggregatorconf

Edit13 the13 loggregator13 file13 with13 the13 following13 contents13

13 Input13 secon13 defines13 which13 log13 records13 to13 receive13

13 port13 and13 type13 reflect13 method13 selected13 for13 Cloud13 Foundry13 to13 broadcast13 the13 records13

input

tcp

port =gt 5000

type =gt syslog

13 Filter13 secon13 ndash13 transforms13 the13 data13

filter

Add a field to identify the records as Cloud Foundry based

mutate

add_field =gt format =gt cf

13 Output13 secon13 ndash13 sends13 the13 data13 to13 its13 desnaon(s)13

output

13 Use13 the13 elascsearch13 plugin13 to13 send13 the13 data13 to13 the13 elascsearch13 instance13

13 elasticsearch

hosts =gt localhost

stdout codec =gt rubydebug

13 End13 of13 Config13

Save13 the13 file13 Verify13 the13 format13 of13 the13 file13 is13 acceptable13 by13 running13 the13 following13 commands13

[rootelkforcase]$ cd homelogstashlogstash-233

[rootelkforcase]$ binlogstash agent --configtest -f configloggregatorconf

Page of 20 71 Page of 20 71

Configuraon13 is13 OK13

Start13 Logstash13 Aaer13 you13 have13 wricen13 and13 verified13 the13 Logstash13 configuraon13 you13 can13 start13 Logstash13 to13 process13 records13 from13 the13 Bluemix13 apps13 (via13 Cloud13 Foundry)13 Run13 the13 following13 as13 root13 user13

[rootelkforcase]$ cd homelogstashlogstash-233

[rootelkforcase]$ binlogstash agent -f configloggregatorconf

Logstash13 is13 now13 started13

Monitor13 Logstash13 processing13 Once13 Logstash13 is13 running13 and13 log13 records13 are13 being13 produced13 by13 the13 Bluemix13 apps13 you13 can13 monitor13 progress13 in13 the13 same13 window13 that13 was13 used13 to13 start13 Logstash13 Output13 is13 similar13 to13 the13 following13

13

The13 log13 records13 being13 broadcast13 from13 the13 Bluemix13 apps13 are13 now13 sent13 to13 Elascsearch13 from13 the13 Logstash13 output13 configuraon13 Records13 are13 sent13 in13 real13 me13 If13 Logstash13 is13 not13 running13 while13 log13 records13 are13 being13 produced13 then13 these13 records13 will13 not13 be13 processed13 by13 Logstash13 and13 will13 not13 be13 stored13 in13 Elascsearch13

Page of 21 71 Page of 21 71

Use13 ELK13 The13 log13 records13 from13 the13 Bluemix13 apps13 are13 now13 being13 stored13 and13 indexed13 by13 the13 Elascsearch13 component13 This13 data13 can13 be13 accessed13 via13 the13 Kibana13 applicaon13

Kibana13 Discover13 page13 Using13 a13 web13 browser13 either13 directly13 from13 a13 client13 or13 using13 the13 VNC13 client13 access13 Kibana13 using13 the13 publicIPaddress13 of13 the13 ELK13 virtual13 server13

hcpltpublicIPaddressgt560113

No13 userpassword13 credenals13 are13 required13

Kibana13 defaults13 to13 a13 search13 of13 all13 records13 ()13 in13 the13 last13 1513 minutes13 and13 returns13 the13 results13

13

More13 complex13 searches13 are13 possible13 by13 changing13 the13 me13 filter13 and13 supplying13 search13 queries13

Change13 the13 0me13 filter13 To13 change13 the13 me13 ranges13 click13 the13 clock13 symbol13 in13 the13 top-shy‐right13 of13 the13 Kibana13 Discover13 page13

13

You13 will13 see13 the13 following13 opons13

13

Page of 22 71 Page of 22 71

A13 variety13 of13 quick13 me13 ranges13 can13 be13 selected13 or13 you13 can13 switch13 to13 relave13 or13 absolute13 me13 ranges13

Quick13 me13 ranges13 allow13 switching13 between13 last13 X13 minutes13 to13 last13 Y13 hours13 or13 Z13 days13 etc13

Relave13 me13 ranges13 allow13 more13 specific13 versions13 of13 the13 Quick13 opons13 from13 specific13 minuteshoursdays13 to13 now13

13

Absolute13 me13 ranges13 specify13 the13 start13 and13 end13 mes13 for13 the13 searches13

13

Custom13 search13 queries13 You13 can13 use13 the13 text13 window13 of13 the13 Discover13 page13 to13 search13 for13 matches13 within13 the13 log13 records13 You13 can13 use13 Apache13 Lucene13 or13 Elascsearch13 query13 DSL13 syntax13

13

Page of 23 71 Page of 23 71

Saved13 searches13 You13 can13 save13 any13 custom13 search13 syntax13 and13 choice13 of13 me13 range13 for13 future13 ease13 of13 use13 Any13 relave13 or13 absolute13 me13 frame13 is13 saved13 as13 part13 of13 the13 search13 13

To13 save13 a13 search13

1 Set13 up13 and13 run13 the13 required13 syntax13 and13 me13 frame13

13

2 Click13 the13 Save13 Search13 icon13

13

3 Type13 in13 a13 name13 for13 the13 saved13 search13

Page of 24 71 Page of 24 71

13

7 To13 delete13 or13 update13 saved13 searches13 click13 13 manage13 saved13 searches13

13

Use13 the13 edit13 icons13 to13 make13 any13 changes13

Page of 26 71 Page of 26 71

13

Build13 simple13 Kibana13 histograms13 Kibana13 allows13 you13 to13 build13 charts13 and13 dashboards13 based13 on13 the13 search13 capability13 offered13 by13 the13 Discover13 page13 Saved13 searches13 can13 be13 used13 as13 the13 basis13 for13 charts13 and13 dashboards13 built13 using13 those13 charts13 To13 build13 a13 simple13 histogram13

1 Select13 the13 Visualize13 page13

13

2 Select13 Ver0cal13 Bar13 Chart13

13

3 To13 use13 a13 saved13 search13 click13 From13 a13 saved13 search13

Page of 27 71 Page of 27 71

13

4 Select13 a13 previously13 saved13 search13

13

5 Select13 to13 edit13 the13 x-shy‐axis

6 Edit13 the13 fields13 with13 informaon13 about13 the13 aggregaon13 field13 interval13 and13 custom13 label13

Page of 28 71 Page of 28 71

13

Page of 29 71 Page of 29 71

7 Click13 the13 green13 arrow13 The13 histogram13 is13 created13 and13 run13 based13 on13 the13 saved13 search

13

8 To13 save13 the13 histogram13 click13 the13 Save13 Visualiza0on13 icon13 and13 add13 a13 tle13

13

13

9 Click13 Save13

10 To13 view13 the13 visualisaon13 click13 the13 Dashboard13 page13

Page of 30 71 Page of 30 71

13

11 Click13 the13 Add13 Visualiza0on13 icon13

13

12 Select13 the13 visiualizaon13 that13 you13 just13 created13

13

Page of 31 71 Page of 31 71

13

13 To13 save13 the13 dashboard13 click13 the13 Save13 Dashboard13 icon13

Page of 32 71 Page of 32 71

13

14 Add13 a13 tle13 and13 select13 whether13 to13 store13 the13 selected13 me13 frame13 with13 the13 dashboard13

13

15 Click13 Save13

16 You13 can13 edit13 the13 shape13 and13 number13 of13 charts13 and13 save13 the13 dashboard13

17 To13 re-shy‐run13 the13 dashboard13 select13 the13 Dashboard13 page13 and13 click13 the13 Load13 Saved13 Dashboard13 icon13

Page of 33 71 Page of 33 71

13

18 Select13 the13 saved13 dashboard13 from13 the13 list13

13

19 The13 dashboard13 runs13

13

Page of 34 71 Page of 34 71

Page of 35 71 Page of 35 71

Aler0ng13 with13 ELK13 ELK13 examines13 the13 log13 records13 in13 process13 to13 send13 alerts13 to13 third-shy‐party13 alerng13 event13 management13 and13 social13 media13 applicaons13 As13 shown13 previously13 in13 the13 ELK13 architecture13 the13 ELK13 strategy13 for13 alerng13 is13 to13 send13 alerts13 from13 the13 Logstash13 component13 directly13 Therefore13 log13 records13 and13 alerts13 are13 sent13 to13 both13 Elascsearch13 and13 third-shy‐party13 tools13 at13 the13 same13 instant13 using13 the13 output13 secon13 of13 the13 Logstash13 configuraon13 Any13 alerts13 are13 generated13 during13 the13 filter13 secon13 processing13 of13 the13 Logstash13 configuraon13

Create13 alerts13 in13 Logstash13 The13 Logstash13 filter13 secon13 is13 used13 to13 trigger13 alerts13 based13 on13 the13 log13 records13 being13 processed13 For13 example13 log13 records13 can13 be13 checked13 for13 errors13 warnings13 crashed13 or13 stopped13 messages13 and13 generate13 an13 alert13

Here13 is13 an13 example13 Logstash13 filter13 code13

13 if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

The13 specific13 tags13 are13 set13 against13 the13 log13 records13 to13 trigger13 the13 alert13 later13 in13 the13 configuraon13 in13 the13 output13 secon13

Control13 the13 number13 of13 generated13 alerts13 using13 the13 Logstash13 throcle13 plugin13 so13 that13 either13 a13 lot13 of13 instances13 in13 a13 fixed13 me13 period13 are13 required13 before13 an13 alert13 is13 sent13 or13 a13 specific13 number13 of13 instances13 triggers13 the13 alert13

Page of 36 71 Page of 36 71

Edit13 the13 plugin13 parameters13 to13 meet13 system13 requirements13

13 13 13 13 13 if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 10

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle13

after_count =gt 20

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 30

period =gt 1200

key =gt servicename_alarm_warning

Page of 37 71 Page of 37 71

add_tag =gt servicename_throttled

13 13 13 13 13 13 13 13 13

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 40

period =gt 1500

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

Enrich13 alerts13 with13 message13 detail13 Some13 of13 the13 third-shy‐party13 integraons13 benefit13 from13 having13 components13 of13 the13 raw13 log13 record13 sent13 in13 addion13 to13 the13 raw13 record13 or13 fields13 in13 the13 record13 edited13 transformed13 or13 translated13

Grok13 Logstash13 allows13 the13 log13 record13 to13 be13 broken13 up13 into13 logical13 secons13 and13 each13 secon13 stored13 in13 a13 new13 parameter13 using13 the13 grok13 parameter13 in13 the13 filter13 secon13 Here13 is13 an13 example13 grok13

13 grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

To13 build13 the13 grok13 statement13 Logstash13 offers13 an13 online13 tool13 at13 hcpsgrokdebugherokuappcom13

Translate13 If13 specific13 fields13 in13 the13 log13 record13 are13 coded13 (for13 example13 the13 Bluemix13 app13 name13 is13 oaen13 not13 sent13 in13 the13 syslog13 messages13 instead13 the13 GUID13 is13 sent13 which13 is13 not13 useful)13 then13 the13 filter13 secon13 translate13 plugin13 can13 be13 used13 to13 add13 new13 fields13 by13 holding13 the13 decoded13 versions13 of13 those13 values13

13 translate

field =gt data1

destination =gt data1

override =gt true

Page of 38 71 Page of 38 71

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

dc50f6e4-c052-436a-bf27-95ba392c4228 monverifyservicemanagement

0c677e82-5bfc-4e38-9155-e416249e3ee6 ms-catalog-postpyretic-pomiculture

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

However13 the13 translate13 plugin13 is13 not13 included13 with13 the13 Logstash13 installaon13 In13 the13 next13 secon13 I13 will13 13 describe13 how13 to13 add13 new13 plugins13 to13 the13 Logstash13 instance13

Add13 custom13 plugins13 to13 Logstash13 As13 an13 open13 source13 tool13 Logstash13 encourages13 users13 to13 develop13 content13 for13 all13 users13 to13 share13 Some13 of13 the13 plugins13 that13 were13 created13 by13 our13 community13 include13 a13 translaon13 plugin13 an13 output13 Slack13 plugin13 and13 an13 output13 syslog13 plugin13 Letrsquos13 look13 at13 how13 to13 add13 these13 plugins13 to13 your13 environment13

Filter13 translate13 plugin13 To13 add13 the13 translate13 plugin13

1 Download13 the13 plugin13

2 Copy13 the13 plugin13 file13 logstash-shy‐filter-shy‐translate-shy‐214gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-filter-translate 214

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Output13 Slack13 plugin13 To13 add13 the13 Slack13 plugin13

Page of 39 71 Page of 39 71

1 Download13 the13 plugin13 13

2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐slack-shy‐014gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-output-slack 014

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Output13 syslog13 plugin13 To13 add13 the13 syslog13 plugin13

1 Download13 the13 plugin13 13

2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐syslog-shy‐214gem13 to13 the13 ELK13 Virtual13 Server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-output-syslog 214

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Integrate13 Logstash13 with13 Slack13 To13 send13 generated13 alerts13 to13 Slack13 complete13 these13 steps13

1 Create13 a13 Slack13 integraon13 13

2 Use13 the13 Slack13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 Slack13 integraon13

113 Login13 to13 Slack13 and13 go13 to13 the13 Apps13 page13 13

Page of 40 71 Page of 40 71

13

313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13

13

A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13

13

Page of 42 71 Page of 42 71

The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13

13

The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13

13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

slack

url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5

channel =gt cloud-operations-mon

username =gt nick_cawood

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13

Page of 43 71 Page of 43 71

Page of 44 71 Page of 44 71

In13 Slack13 the13 alerts13 look13 like13 this13

13

Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13

1 Create13 a13 PagerDuty13 integraon13 13

2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13

1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13

13

2 Click13 Add13 New13 Service13

13

313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13

Page of 45 71 Page of 45 71

13

413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13

513 Click13 Add13 Service13

13

The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13

Page of 46 71 Page of 46 71

13

The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt be7eb1ea9c224018923e818d3e891d0b

incident_key =gt logstashhosttype

Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13

Page of 47 71 Page of 47 71

13

Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13

1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13

2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13

3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13

4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13

13 13 http

url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo

http_method =gt post

format =gt json

For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13

13 13 syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

Page of 48 71 Page of 48 71

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13

13

The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13

if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2

This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13

Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13

Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13

Logstash13 severity Shown13 in13 Omnibus13 as

alert Indeterminate

crical Warning

debug 7

emergency Clear

error Minor

informaonal 6

noce Crical

warning Major

Page of 49 71 Page of 49 71

13

Page of 50 71 Page of 50 71

Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13

input

tcp

port =gt 5000

type =gt syslog

filter

mutate

add_field =gt format =gt cf

grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

translate

field =gt data1

destination =gt data1

override =gt true

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

=== Check Alarm Criteria ===

Standard Key Words

if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

=== Alarm Exclusions ===

if [message] =~ (i)sometexthere

mutate remove_tag =gt servicename_alarm

Throttle alarms so we do not hit PD rate limits

Page of 52 71 Page of 52 71

if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 5

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle

after_count =gt 5

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 5

period =gt 1200

key =gt servicename_alarm_warning

add_tag =gt servicename_throttled

Page of 53 71 Page of 53 71

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

output

elasticsearch

hosts =gt localhost

stdout codec =gt rubydebug

if servicename_alarm_err in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

Page of 54 71 Page of 54 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 55 71 Page of 55 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

Page of 56 71 Page of 56 71

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert err message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_error in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

Page of 57 71 Page of 57 71

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 58 71 Page of 58 71

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 59 71 Page of 59 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert error message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

Page of 60 71 Page of 60 71

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

Page of 61 71 Page of 61 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 62 71 Page of 62 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt x

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert fatal message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

Page of 63 71 Page of 63 71

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

Page of 64 71 Page of 64 71

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 65 71 Page of 65 71

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt heavy_exclamation_mark

Page of 66 71 Page of 66 71

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert WARNING message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 67 71 Page of 67 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

Page of 68 71 Page of 68 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 69 71 Page of 69 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

Page of 70 71 Page of 70 71

Page of 71 71 Page of 71 71

Page 3: How to integrate the Elasticsearch Logstash Kibana (ELK ... · How to integrate the Elasticsearch Logstash Kibana (ELK) log analytics stack into IBM Bluemix Nick Cawood IBM Cloud

Introduc0on13 IBMreg13 Bluemixreg13 IBMrsquos13 PlaRorm13 as13 a13 Service13 (PaaS)13 allows13 you13 to13 rapidly13 build13 and13 deploy13 an13 applicaon13 Applicaons13 can13 run13 in13 a13 nave13 or13 a13 hybrid13 Bluemix13 environment13 13 13

This13 document13 describes13 the13 integraon13 of13 the13 Elascsearch13 Logstash13 Kibana13 (ELK)13 log13 analycs13 open13 source13 stack13 into13 the13 Bluemix13 environment13 how13 to13 collect13 logs13 from13 Bluemix13 applicaons13 (apps)13 and13 send13 alerts13 from13 those13 log13 records13 to13 selected13 alertevent13 noficaon13 targets13 (PagerDuty13 Slack13 and13 IBM13 Netcool13 Omnibus)13

This13 document13 describes13 how13 to13 provision13 Bluemix13 virtual13 servers13 in13 the13 Bluemix13 environment13 install13 the13 ELK13 stack13 on13 a13 Bluemix13 virtual13 server13 and13 configure13 the13 ELK13 stack13 to13 integrate13 with13 other13 components13 deployed13 on13 the13 Bluemix13 environment13

The13 steps13 to13 integrate13 the13 ELK13 stack13 into13 the13 Cloud13 Architecture13 and13 Soluon13 Engineering13 (CASE)13 environment13 are13 used13 as13 the13 basis13 for13 this13 document13 13 For13 reference13 both13 the13 ELK13 architecture13 within13 the13 CASE13 environment13 and13 the13 whole13 CASE13 environment13 are13 defined13 using13 system13 architecture13 diagrams13 as13 well13 as13 diagrams13 defining13 the13 ELK13 structure13 and13 ELK13 integraon13 into13 Bluemix13 architecture13

What13 is13 Elas0csearch13 Logstash13 Kibana13 Elascsearch13 Logstash13 Kibana13 is13 an13 open13 source13 stack13 of13 three13 applicaons13 that13 work13 together13 to13 provide13 an13 end-shy‐to-shy‐end13 search13 and13 visualisaon13 plaRorm13 for13 the13 invesgaon13 of13 log13 file13 sources13 in13 real13 me13 Log13 file13 records13 can13 be13 searched13 from13 a13 variety13 of13 log13 sources13 charts13 and13 dashboards13 built13 to13 visualize13 the13 log13 records13 The13 three13 applicaons13 are13 Elascsearch13 Logstash13 and13 Kibana13

Logstash13 Logstash13 is13 an13 applicaon13 that13 allows13 the13 ELK13 stack13 to13 manage13 the13 gathering13 transformaon13 and13 transport13 of13 log13 file13 records13 Logstash13 is13 able13 to13 pull13 records13 from13 remote13 sources13 or13 listen13 for13 records13 on13 specified13 ports13

Logstash13 has13 three13 areas13 of13 funcon13

bull Process13 input13 of13 the13 log13 file13 records13 from13 the13 remote13 source13 You13 can13 configure13 Logstash13 to13 receive13 records13 in13 a13 variety13 of13 methods13 for13 example13 reading13 from13 a13 flat13 file13 or13 from13 a13 TCP13 port13 or13 direct13 from13 a13 remote13 or13 local13 syslog13 These13 methods13 are13 defined13 in13 the13 Logstash13 online13 support13 and13 input13 secon13 of13 the13 Logstash13 conf13 file13 Once13 all13 input13 processing13 is13 completed13 Logstash13 moves13 onto13 the13 next13 funcon13

bull Transform13 and13 enrich13 the13 log13 records13 that13 take13 place13 in13 the13 Logstash13 filter13 secon13 Any13 number13 of13 changes13 to13 the13 format13 (and13 content13 if13 required)13 of13 the13 log13 file13 record13 can13 be13 made13 using13 the13 various13 methods13 provided13 with13 Logstash13 for13 example13 methods13 to13 add13 and13 enrich13 log13 record13 fields13 and13 the13 flagging13 I13 describe13 excluding13 and13 liming13 records13 for13 alerng13 later13 in13 this13 document13 Aaer13 the13 log13 record13 is13 transformed13 Logstash13 processes13 the13 record13

bull Send13 the13 log13 records13 to13 the13 Elascsearch13 applicaon13 from13 the13 output13 secon13 of13 the13 Logstash13 configuraon13 A13 specific13 elascsearch13 method13 is13 included13 with13 the13 Logstash13 instance13 that13 sends13 the13 log13 records13 to13 the13 Elascsearch13 instance13 However13 as13 with13 the13 input13 and13 filter13

Page of 3 71 Page of 3 71

secons13 a13 variety13 of13 methods13 are13 available13 to13 process13 the13 data13 sent13 from13 the13 filter13 secon13 For13 example13 the13 records13 can13 be13 sent13 to13 external13 alerng13 tools13 such13 as13 PagerDuty13 social13 media13 tools13 such13 as13 Slack13 or13 event13 management13 hubs13 like13 IBM13 Netcool13 Omnibus13 via13 output13 methods13 syslog13 or13 hcp13 13

As13 an13 open13 source13 tool13 non-shy‐Logstash13 employees13 develop13 other13 methods13 and13 share13 them13 with13 the13 user13 community13 for13 addion13 to13 the13 library13 of13 methods13 Logstash13 can13 execute13 (examples13 of13 this13 process13 are13 included13 later13 in13 this13 document)13

Elas0csearch13 Elascsearch13 is13 the13 index13 store13 and13 query13 applicaon13 on13 the13 ELK13 stack13 It13 provides13 a13 standard13 full13 text13 search13 facility13 of13 the13 log13 file13 records13 processed13 by13 Logstash13 The13 ELK13 online13 help13 documentaon13 describes13 Elascsearch13 as13 follows13 ldquoIt13 is13 generally13 used13 as13 the13 underlying13 enginetechnology13 that13 powers13 applicaons13 that13 have13 complex13 search13 features13 and13 requirementsrdquo13

Kibana13 Kibana13 is13 the13 user13 interface13 for13 the13 ELK13 stack13 Kibana13 integrates13 with13 Elascsearch13 to13 visualise13 the13 results13 of13 searches13 and13 allow13 the13 creaon13 of13 charts13 and13 dashboards13 based13 on13 the13 searches13 The13 searches13 charts13 and13 dashboard13 can13 visualise13 real-shy‐me13 or13 historical13 log13 records13 Users13 access13 Kibana13 through13 an13 hcp13 URL13 from13 a13 client13 web13 browser13

ELK13 System13 Architecture13 The13 ELK13 stack13 can13 be13 described13 as13 follows13

Page of 4 71 Page of 4 71

13

Page of 5 71 Page of 5 71

Service13 Management13 architecture13 The13 IBM13 Cloud13 Architecture13 Center13 created13 a13 standard13 reference13 architecture13 for13 how13 to13 set13 up13 your13 cloud13 environemnt13 to13 handle13 incident13 management13 13

View13 the13 standard13 Service13 Management13 architecure13

13 13

Log13 Monitoring13 System13 architecture13 The13 following13 diagram13 shows13 a13 standard13 architecture13 for13 the13 log-shy‐monitoring13 system13 described13 in13 this13 tutorial13 13

13

Bluemix13 environment13 system13 architecture13 The13 ELK13 stack13 can13 be13 installed13 into13 a13 variety13 of13 environments13 When13 using13 Bluemix13 the13 following13 figure13 shows13 what13 the13 environment13 might13 look13 like13

Page of 6 71 Page of 6 71

13

The13 Bluemix13 cloud13 plaRorm13 offers13 a13 Public13 network13 and13 access13 to13 US13 South13 and13 UK13 environments13 From13 all13 these13 environments13 third-shy‐party13 cloud-shy‐based13 soluons13 are13 also13 accessible13

The13 ELK13 stack13 is13 installed13 in13 the13 Bluemix13 United13 Kingdom13 environment13 because13 this13 is13 the13 environment13 where13 Bluemix13 has13 virtual13 servers13 available13 for13 provisioning13 Other13 applicaons13 in13 the13 CASE13 project13 were13 also13 installed13 on13 virtual13 servers13 in13 the13 Bluemix13 United13 Kingdom13 region13

Bluemix13 apps13 were13 used13 to13 create13 a13 system13 to13 be13 monitored13 by13 ELK13 These13 apps13 were13 hosted13 in13 the13 Bluemix13 US13 South13 environment13 using13 Bluemix13 microservices13 which13 produce13 logs13 during13 normal13 operaon13 and13 during13 crashes13 and13 outages13

Through13 the13 Bluemix13 Cloud13 Foundry13 process13 the13 log13 records13 can13 be13 broadcast13 from13 the13 Bluemix13 apps13 so13 Logstash13 can13 receive13 them13 and13 send13 them13 to13 Elascsearch13 within13 the13 ELK13 stack13 or13 other13 virtual13 servers13 (hosng13 other13 IBM13 or13 third-shy‐party13 products)13 in13 the13 Bluemix13 United13 Kingdom13 environment13 or13 to13 third-shy‐party13 applicaons13 in13 the13 public13 cloud13 (Pager13 Duty13 Slack13 etc)13

Crea0ng13 a13 virtual13 server13 in13 Bluemix13 With13 Bluemix13 you13 can13 provision13 a13 virtual13 server13 in13 the13 cloud13 to13 install13 standalone13 or13 integrated13 system13 components13 Not13 all13 Bluemix13 regions13 offer13 this13 service13 all13 the13 me13 due13 to13 beta-shy‐project13 constraints13 In13 this13 arclersquos13 examples13 virtual13 servers13 were13 allowed13 to13 be13 provisioned13 in13 the13 United13 Kingdom13 region13 For13 more13 informaon13 read13 how13 to13 provision13 a13 Bluemix13 virtual13 server13 including13 some13 simple13 setup13 steps13 for13 ease13 of13 use13

Page of 7 71 Page of 7 71

Create13 an13 SSH13 key13 To13 access13 your13 virtual13 server13 once13 it13 is13 provisioned13 allocate13 an13 SSH13 key13 to13 the13 virtual13 server13

To13 generate13 a13 publicprivate13 RSA13 key13 pair13 follow13 these13 steps13

1 In13 a13 UNIX13 or13 Linuxreg13 window13 run13 the13 following13 command13 specifying13 a13 ltuniquekeynamegt13 of13 your13 your13 choice13 and13 a13 password13 or13 passphrase13 (memorize13 the13 password)$13 ssh-shy‐keygen13 -shy‐t13 rsa13 -shy‐f13 ltuniquekeynamegt13

2 Enter13 the13 passphrase13 (empty13 for13 no13 passphrase)13

3 Enter13 the13 same13 passphrase13 again13

Your13 idenficaon13 has13 been13 saved13 in13 ltuniquekeynamegt13

Your13 public13 key13 has13 been13 saved13 in13 ltuniquekeynamegtpub13

The13 key13 fingerprint13 is13

d6268464506525930ce4632c693c203c13 ltusergtltservergt13

The13 keys13 randomart13 image13 is13

+--[ RSA 2048]----+

|o ++=o |

| E o ooo |

| |

| + o |

| S o |

| o |

| |

| |

| |

+-----------------+

Two13 keys13 are13 created13 public13 and13 private13 The13 public13 key13 contents13 are13 used13 to13 create13 a13 virtual13 server13 with13 a13 public13 IP13 address13 The13 private13 key13 is13 used13 to13 access13 the13 virtual13 server13 from13 a13 client13 machine13

Page of 8 71 Page of 8 71

Provisioning13 a13 virtual13 server13 1 A13 Bluemix13 account13 is13 required13 for13 this13 process13 Sign13 up13 here13 and13 log13 in13 Make13 sure13 your13

environment13 is13 correct13 For13 example13 if13 you13 are13 in13 the13 United13 Kingdom13 select13 the13 United13 Kingdom13 region13 in13 the13 Bluemix13 console13

13

13

2 Add13 a13 virtual13 server13 to13 your13 Bluemix13 environment13 Click13 Run13 Virtual13 Servers13

13

3 Select13 an13 OS13 image13 from13 the13 EXISTING13 list13 For13 ELK13 select13 Centos13 OS13 (use13 latest13 available13 version)13

Page of 9 71 Page of 9 71

4 Name13 the13 virtual13 server13

5 Make13 sure13 Assign13 public13 IP13 address13 is13 checked13

13

6 Click13 Add13 Key13 to13 add13 the13 SSH13 key13 you13 created13 previously13 Copy13 the13 contents13 of13 the13 public13 key13 into13 the13 Public13 Key13 to13 import13 field13

Page of 10 71 Page of 10 71

13

7 Click13 OK13 Then13 click13 CREATE13

The13 virtual13 server13 is13 provisioned13 and13 added13 to13 the13 Bluemix13 dashboard13 under13 VIRTUAL13 SERVERS13

Page of 11 71 Page of 11 71

13

Set13 up13 the13 virtual13 server13

Log13 on13 to13 the13 virtual13 server13 Now13 you13 can13 log13 on13 to13 the13 virtual13 server13 and13 add13 new13 users13 and13 programs13 to13 make13 it13 easier13 to13 access13 the13 virtual13 server13

To13 access13 the13 server13 go13 to13 the13 UNIX13 or13 Linux13 window13 that13 you13 used13 to13 create13 the13 SSH13 key13 In13 the13 directory13 where13 the13 ltuniquekeynamegt13 files13 exist13 run13 the13 following13 command13 referencing13 the13 SSH13 keyname13 the13 public13 IP13 address13 Bluemix13 allocated13 for13 the13 virtual13 server13 and13 the13 password13 for13 the13 SSH13 key13

[userserver ~]$ ssh -i ltuniquekeynamegt ibmcloudltpublicIPaddressgt

Enter13 the13 passphrase13 for13 the13 key13 ltuniquekeynamegt13

Last login Tue Jul 5 125546 2016 from 108-210-96-231lightspeedrlghncsbcglobalnet

[ibmcloudelkforcase ~]$

ibmcloud13 is13 the13 default13 user13 for13 the13 Bluemix13 virtual13 servers13 and13 allows13 SSH13 to13 access13 the13 server13

Acces13 the13 root13 user13 To13 access13 the13 root13 user13 log13 on13 as13 ibmcloud13 user13 and13 run13 the13 following13 command13

[ibmcloudelkforcase ~]$ sudo bash

[rootelkforcase ibmcloud]

The13 root13 user13 can13 be13 used13 to13 add13 programs13 to13 the13 server13 and13 create13 and13 edit13 users13 as13 required13 to13 access13 applicaons13 installed13 on13 the13 virtual13 server13

Page of 12 71 Page of 12 71

Copy13 files13 to13 the13 virtual13 server13 To13 move13 installaon13 files13 to13 the13 server13 go13 to13 the13 UNIX13 or13 Linux13 window13 you13 used13 to13 create13 the13 SSH13 key13 and13 the13 directory13 where13 the13 ltuniquekeynamegt13 files13 exist13 Move13 the13 installaon13 files13 to13 the13 same13 locaon13

Run13 the13 following13 command13 referencing13 the13 SSH13 keyname13 the13 public13 IP13 address13 Bluemix13 allocated13 for13 the13 virtual13 server13 and13 the13 password13 for13 the13 SSH13 key13

[userserver ~]$ scp ndashi ltuniquekeynamegt ltINSTALLATIONFILENAMEgt ibmcloudltpublicIPaddressgthomeibmcloud

Enter13 the13 passphrase13 for13 key13 cloudkey313

This13 copies13 the13 file13 using13 the13 ibmcloud13 user13 to13 the13 homeibmcloud13 directory13 on13 the13 virtual13 server13

Install13 and13 access13 the13 virtual13 server13 via13 VNC13 server13 Using13 the13 ssh13 command13 to13 access13 the13 virtual13 server13 is13 not13 the13 most13 stable13 method13 and13 does13 not13 offer13 X-shy‐Windows13 support13 Therefore13 we13 suggest13 that13 you13 use13 a13 VNC13 client13 to13 access13 the13 virtual13 server13

Installing13 a13 VNC13 client13 should13 be13 done13 as13 the13 root13 user13 Use13 the13 yum13 command13 as13 follows13

[ibmcloudelkforcase ~]$ yum groupinstall GNOME Desktop

[ibmcloudelkforcase ~]$ yum install tigervnc-server

These13 commands13 first13 install13 a13 desktop13 style13 OS13 user13 interface13 and13 then13 a13 VNC13 client13 to13 remotely13 access13 the13 desktop13 style13 OS13

Then13 as13 either13 the13 root13 user13 or13 another13 user13 (example13 below13 uses13 elk)13 run13 the13 following13 command13 to13 create13 the13 VNC13 client13 connecon13

[elkelkforcase ~]$ vncserver

New elkforcase2 (elk) desktop is elkforcase2

Starting applications specified in homeelkvncxstartup

Log file is homeelkvncelkforcase2log

[elkelkforcase ~]$

Install13 a13 VNC13 client13 on13 the13 client13 machine13 used13 to13 access13 the13 Internet13 and13 (in13 the13 example13 above)13 use13 the13 command13 ltpublicIPadddressgt213 and13 the13 password13 for13 ELK13 to13 access13 the13 virtual13 server13

Using13 a13 VNC13 client13 means13 that13 you13 can13 leave13 long13 duraon13 installaons13 running13 without13 fear13 of13 the13 terminal13 losing13 connecon13 and13 the13 installaons13 failing13

Install13 the13 ELK13 stack13 Download13 the13 source13 soaware13 code13 for13 the13 ELK13 stack13 from13 the13 ELK13 website13 13

At13 the13 me13 of13 wring13 the13 downloaded13 soaware13 files13 were13

Page of 13 71 Page of 13 71

13

Transfer13 the13 files13 to13 the13 virtual13 server13 (using13 the13 scp13 method13 detailed13 above)13

Install13 Elas0csearch13 As13 a13 root13 user13 create13 a13 non-shy‐root13 user13 with13 the13 adduser13 command13

Create13 a13 directory13 to13 install13 Elascsearch13 into13 and13 untar13 the13 contents13 of13 the13 Elascsearch13 install13 file13 into13 that13 directory13

[elkelkforcase]$ cd home

[elkelkforcase]$ mkdir elasticsearch

[elkelkforcase]$ cp homeibmcloudelasticsearch-233targz homeelasticsearch

[elkelkforcase]$ cd elasticsearch

[elkelkforcase]$ gunzip elasticsearch-233targz

[elkelkforcase]$ untar elasticsearch-233tar

Elascsearch13 is13 now13 installed13

Start13 Elas0csearch13 To13 start13 Elascsearch13 go13 to13 the13 bin13 directory13 as13 the13 non-shy‐root13 user13 when13 you13 installed13 it13 and13 run13 the13 elascsearch13 executable13

[elkelkforcase]$ cd homeelasticsearchelasticsearch-233bin

[elkelkforcase]$ elasticsearch amp

[2] 22211

[elkelkforcase]$ [2016-07-05 133857730][INFO ][node] [Arsenal] version[233] pid[22211] build[218bdf12016-05-17T154004Z]

[2016-07-05 133857740][INFO ][node] [Arsenal] initializing

[2016-07-05 133858984][INFO ][plugins] [Arsenal] modules [reindex lang-expression lang-groovy] plugins [] sites []

[2016-07-05 133859146][INFO ][env] [Arsenal] using [1] data paths mounts [[ (rootfs)]] net usable_space [155gb] net total_space [199gb] spins [unknown] types [rootfs]

[2016-07-05 133859146][INFO ][env] [Arsenal] heap size [10156mb] compressed ordinary object pointers [true]

Page of 14 71 Page of 14 71

[2016-07-05 133859147][WARN ][env] [Arsenal] max file descriptors [4096] for elasticsearch process likely too low consider increasing to at least [65536]

[2016-07-05 133903526][INFO ][node] [Arsenal] initialized

[2016-07-05 133903526][INFO ][node] [Arsenal] starting

To13 test13 that13 Elascsearch13 is13 running13 correctly13 open13 the13 browser13 in13 the13 VNC13 client13 and13 run13 localhost920013

13

Install13 Kibana13 ELK13 recommends13 installing13 and13 running13 Kibana13 as13 root13 13

1 As13 root13 user13 create13 an13 installaon13 directory13 and13 untar13 the13 installaon13 file13

[rootelkforcase]$ cd home

[rootelkforcase]$ mkdir kibana

[rootelkforcase]$ cp homeibmcloudkibana-451-linux-x64targz homekibana

[rootelkforcase]$ cd kibana

[rootelkforcase]$ gunzip kibana-451-linux-x64targz

[rootelkforcase]$ untar kibana-451-linux-x64tar

2 Modify13 the13 manifestyml13 configuraon13 file13 Go13 to13 the13 kibana-shy‐451-shy‐linux-shy‐x6413 directory13

[rootelkforcase]$ cd homekibanakibana-451-linux-x64

3 Edit13 the13 manifestyml13 file13 so13 the13 contents13 are13

applicaons13

-shy‐ name13 kibana-shy‐in-shy‐bluemix13

memory13 128M13

host13 kibana-shy‐in-shy‐bluemix13

buildpack13 hcpsgithubcomcloudfoundry-shy‐communitynginx-shy‐buildpackgit13

Page of 15 71 Page of 15 71

4 Edit13 the13 configkibanayml13 file13 so13 the13 contents13 are13

Kibana is served by a back end server This controls which port to use

port 5601

The host to bind the server to

host 0000

If you are running kibana behind a proxy and want to mount it at a path

specify that path here The basePath cant end in a slash

serverbasePath

The maximum payload size in bytes on incoming server requests

servermaxPayloadBytes 1048576

The Elasticsearch instance to use for all your queries

elasticsearchurl httplocalhost9200

preserve_elasticsearch_host true will send the hostname specified in `elasticsearch` If you set it to false

then the host you use to connect to this Kibana instance will be sent

elasticsearchpreserveHost true

Kibana uses an index in Elasticsearch to store saved searches visualizations

and dashboards It will create a new index if it doesnt already exist

kibanaindex kibana

The default application to load

kibanadefaultAppId discover

If your Elasticsearch is protected with basic auth these are the user credentials

used by the Kibana server to perform maintenance on the kibana_index at startup Your

Kibana users will still need to authenticate with Elasticsearch (which is proxied through

the Kibana server)

Page of 16 71 Page of 16 71

elasticsearchusername user

elasticsearchpassword pass

SSL for outgoing requests from the Kibana Server to the browser (PEM formatted)

serversslcert pathtoyourservercrt

serversslkey pathtoyourserverkey

Optional setting to validate that your Elasticsearch backend uses the same key files (PEM formatted)

elasticsearchsslcert pathtoyourclientcrt

elasticsearchsslkey pathtoyourclientkey

If you need to provide a CA certificate for your Elasticsearch instance put

the path of the pem file here

elasticsearchsslca pathtoyourCApem

Set to false to have a complete disregard for the validity of the SSL

certificate

elasticsearchsslverify true

Time in milliseconds to wait for elasticsearch to respond to pings defaults to

request_timeout setting

elasticsearchpingTimeout 1500

Time in milliseconds to wait for responses from the back end or elasticsearch

This must be gt 0

elasticsearchrequestTimeout 30000

Time in milliseconds for Elasticsearch to wait for responses from shards

Set to 0 to disable

elasticsearchshardTimeout 0

Time in milliseconds to wait for Elasticsearch at Kibana startup before retrying

elasticsearchstartupTimeout 5000

elasticsearchstartupTimeout 30000

Page of 17 71 Page of 17 71

Set the path to where you would like the process id file to be created

pidfile varrunkibanapid

If you would like to send the log output to a file you can set the path below

loggingdest stdout

Set this to true to suppress all logging output

loggingsilent false

Set this to true to suppress all logging output except for error messages

loggingquiet false

Set this to true to log all events including system usage information and all requests

loggingverbose false

Kibana13 is13 now13 installed13

Start13 Kibana13 To13 start13 Kibana13 go13 to13 the13 bin13 directory13 and13 run13 the13 kibana13 executable13

[rootelkforcase]$

[rootelkforcase]$ cd homekibanakibana-451-linux-x64bin

[rootelkforcase]$ kibana amp

[1] 22283

[rootelkforcase] log [095900542] [info][status][pluginkibana] Status changed from uninitialized to green - Ready

log [095900600] [info][status][pluginelasticsearch] Status changed from uninitialized to yellow - Waiting for Elasticsearch

log [095900689] [info][status][pluginkbn_vislib_vis_types] Status changed from uninitialized to green - Ready

log [095900717] [info][status][pluginmarkdown_vis] Status changed from uninitialized to green - Ready

log [095900721] [info][status][pluginmetric_vis] Status changed from uninitialized to green - Ready

log [095900814] [info][status][pluginspyModes] Status changed from uninitialized to green - Ready

log [095900934] [info][status][pluginstatusPage] Status changed from uninitialized to green - Ready

Page of 18 71 Page of 18 71

log [095900941] [info][status][plugintable_vis] Status changed from uninitialized to green - Ready

log [095901067] [info][listening] Server running at http00005601

log [095906083] [info][status][pluginelasticsearch] Status changed from yellow to yellow - No existing Kibana index found

log [095908942] [info][status][pluginelasticsearch] Status changed from yellow to green - Kibana index ready

Kibana13 is13 now13 started13

Broadcast13 Bluemix13 app13 log13 records13 Bluemix13 apps13 (which13 are13 shown13 in13 the13 US13 South13 Bluemix13 region13 of13 the13 System13 Architecture13 diagram)13 will13 produce13 logs13 in13 a13 syslog13 rfc542413 format13 using13 a13 Bluemix13 tool13 called13 Loggregator13

For13 Logstash13 to13 receive13 these13 logs13 each13 Bluemix13 app13 needs13 to13 broadcast13 these13 logs13 To13 do13 that13 run13 the13 following13 commands13 on13 each13 Bluemix13 app13 using13 the13 public13 IP13 address13 of13 the13 ELK13 virtual13 server13 the13 ELK13 tag13 to13 trace13 the13 process13 and13 insert13 the13 name13 of13 each13 app13 in13 turn13

cf cups elk -l syslogltpublicIPaddress5000 cf bind-service ltmyappgt elk

cf restage ltmyappgt

Install13 Logstash13 To13 install13 Logstash13 as13 root13 user13 create13 a13 directory13 for13 the13 installaon13 and13 untar13 the13 installaon13 file13 there13

13 13 [rootelkforcase]$ cd home

[rootelkforcase]$ mkdir logstash

[rootelkforcase]$ cp homeibmcloudlogstash-233targz homelogstash

[rootelkforcase]$ cd logstash

[rootelkforcase]$ gunzip logstash-233targz

[rootelkforcase]$ untar logstash-233tar

Write13 Logstash13 configura0on13 Logstash13 requires13 a13 configuraon13 file13 to13 be13 wricen13 to13 receive13 transform13 and13 output13 the13 log13 records13 to13 Elascsearch13 This13 secon13 explains13 the13 basic13 requirements13 of13 processing13 Bluemix13 app13 log13 records13 into13 Elascsearch13 (more13 complex13 configuraons13 are13 detailed13 later13 in13 this13 document13 for13 the13 integraons13 with13 third-shy‐party13 tools)13

Create13 a13 conf13 file13 as13 root13 user13

[rootelkforcase]$ cd homelogstashlogstash-233

[rootelkforcase]$ mkdir config

Page of 19 71 Page of 19 71

[rootelkforcase]$ cd config

[rootelkforcase]$ touch loggregatorconf

Edit13 the13 loggregator13 file13 with13 the13 following13 contents13

13 Input13 secon13 defines13 which13 log13 records13 to13 receive13

13 port13 and13 type13 reflect13 method13 selected13 for13 Cloud13 Foundry13 to13 broadcast13 the13 records13

input

tcp

port =gt 5000

type =gt syslog

13 Filter13 secon13 ndash13 transforms13 the13 data13

filter

Add a field to identify the records as Cloud Foundry based

mutate

add_field =gt format =gt cf

13 Output13 secon13 ndash13 sends13 the13 data13 to13 its13 desnaon(s)13

output

13 Use13 the13 elascsearch13 plugin13 to13 send13 the13 data13 to13 the13 elascsearch13 instance13

13 elasticsearch

hosts =gt localhost

stdout codec =gt rubydebug

13 End13 of13 Config13

Save13 the13 file13 Verify13 the13 format13 of13 the13 file13 is13 acceptable13 by13 running13 the13 following13 commands13

[rootelkforcase]$ cd homelogstashlogstash-233

[rootelkforcase]$ binlogstash agent --configtest -f configloggregatorconf

Page of 20 71 Page of 20 71

Configuraon13 is13 OK13

Start13 Logstash13 Aaer13 you13 have13 wricen13 and13 verified13 the13 Logstash13 configuraon13 you13 can13 start13 Logstash13 to13 process13 records13 from13 the13 Bluemix13 apps13 (via13 Cloud13 Foundry)13 Run13 the13 following13 as13 root13 user13

[rootelkforcase]$ cd homelogstashlogstash-233

[rootelkforcase]$ binlogstash agent -f configloggregatorconf

Logstash13 is13 now13 started13

Monitor13 Logstash13 processing13 Once13 Logstash13 is13 running13 and13 log13 records13 are13 being13 produced13 by13 the13 Bluemix13 apps13 you13 can13 monitor13 progress13 in13 the13 same13 window13 that13 was13 used13 to13 start13 Logstash13 Output13 is13 similar13 to13 the13 following13

13

The13 log13 records13 being13 broadcast13 from13 the13 Bluemix13 apps13 are13 now13 sent13 to13 Elascsearch13 from13 the13 Logstash13 output13 configuraon13 Records13 are13 sent13 in13 real13 me13 If13 Logstash13 is13 not13 running13 while13 log13 records13 are13 being13 produced13 then13 these13 records13 will13 not13 be13 processed13 by13 Logstash13 and13 will13 not13 be13 stored13 in13 Elascsearch13

Page of 21 71 Page of 21 71

Use13 ELK13 The13 log13 records13 from13 the13 Bluemix13 apps13 are13 now13 being13 stored13 and13 indexed13 by13 the13 Elascsearch13 component13 This13 data13 can13 be13 accessed13 via13 the13 Kibana13 applicaon13

Kibana13 Discover13 page13 Using13 a13 web13 browser13 either13 directly13 from13 a13 client13 or13 using13 the13 VNC13 client13 access13 Kibana13 using13 the13 publicIPaddress13 of13 the13 ELK13 virtual13 server13

hcpltpublicIPaddressgt560113

No13 userpassword13 credenals13 are13 required13

Kibana13 defaults13 to13 a13 search13 of13 all13 records13 ()13 in13 the13 last13 1513 minutes13 and13 returns13 the13 results13

13

More13 complex13 searches13 are13 possible13 by13 changing13 the13 me13 filter13 and13 supplying13 search13 queries13

Change13 the13 0me13 filter13 To13 change13 the13 me13 ranges13 click13 the13 clock13 symbol13 in13 the13 top-shy‐right13 of13 the13 Kibana13 Discover13 page13

13

You13 will13 see13 the13 following13 opons13

13

Page of 22 71 Page of 22 71

A13 variety13 of13 quick13 me13 ranges13 can13 be13 selected13 or13 you13 can13 switch13 to13 relave13 or13 absolute13 me13 ranges13

Quick13 me13 ranges13 allow13 switching13 between13 last13 X13 minutes13 to13 last13 Y13 hours13 or13 Z13 days13 etc13

Relave13 me13 ranges13 allow13 more13 specific13 versions13 of13 the13 Quick13 opons13 from13 specific13 minuteshoursdays13 to13 now13

13

Absolute13 me13 ranges13 specify13 the13 start13 and13 end13 mes13 for13 the13 searches13

13

Custom13 search13 queries13 You13 can13 use13 the13 text13 window13 of13 the13 Discover13 page13 to13 search13 for13 matches13 within13 the13 log13 records13 You13 can13 use13 Apache13 Lucene13 or13 Elascsearch13 query13 DSL13 syntax13

13

Page of 23 71 Page of 23 71

Saved13 searches13 You13 can13 save13 any13 custom13 search13 syntax13 and13 choice13 of13 me13 range13 for13 future13 ease13 of13 use13 Any13 relave13 or13 absolute13 me13 frame13 is13 saved13 as13 part13 of13 the13 search13 13

To13 save13 a13 search13

1 Set13 up13 and13 run13 the13 required13 syntax13 and13 me13 frame13

13

2 Click13 the13 Save13 Search13 icon13

13

3 Type13 in13 a13 name13 for13 the13 saved13 search13

Page of 24 71 Page of 24 71

13

7 To13 delete13 or13 update13 saved13 searches13 click13 13 manage13 saved13 searches13

13

Use13 the13 edit13 icons13 to13 make13 any13 changes13

Page of 26 71 Page of 26 71

13

Build13 simple13 Kibana13 histograms13 Kibana13 allows13 you13 to13 build13 charts13 and13 dashboards13 based13 on13 the13 search13 capability13 offered13 by13 the13 Discover13 page13 Saved13 searches13 can13 be13 used13 as13 the13 basis13 for13 charts13 and13 dashboards13 built13 using13 those13 charts13 To13 build13 a13 simple13 histogram13

1 Select13 the13 Visualize13 page13

13

2 Select13 Ver0cal13 Bar13 Chart13

13

3 To13 use13 a13 saved13 search13 click13 From13 a13 saved13 search13

Page of 27 71 Page of 27 71

13

4 Select13 a13 previously13 saved13 search13

13

5 Select13 to13 edit13 the13 x-shy‐axis

6 Edit13 the13 fields13 with13 informaon13 about13 the13 aggregaon13 field13 interval13 and13 custom13 label13

Page of 28 71 Page of 28 71

13

Page of 29 71 Page of 29 71

7 Click13 the13 green13 arrow13 The13 histogram13 is13 created13 and13 run13 based13 on13 the13 saved13 search

13

8 To13 save13 the13 histogram13 click13 the13 Save13 Visualiza0on13 icon13 and13 add13 a13 tle13

13

13

9 Click13 Save13

10 To13 view13 the13 visualisaon13 click13 the13 Dashboard13 page13

Page of 30 71 Page of 30 71

13

11 Click13 the13 Add13 Visualiza0on13 icon13

13

12 Select13 the13 visiualizaon13 that13 you13 just13 created13

13

Page of 31 71 Page of 31 71

13

13 To13 save13 the13 dashboard13 click13 the13 Save13 Dashboard13 icon13

Page of 32 71 Page of 32 71

13

14 Add13 a13 tle13 and13 select13 whether13 to13 store13 the13 selected13 me13 frame13 with13 the13 dashboard13

13

15 Click13 Save13

16 You13 can13 edit13 the13 shape13 and13 number13 of13 charts13 and13 save13 the13 dashboard13

17 To13 re-shy‐run13 the13 dashboard13 select13 the13 Dashboard13 page13 and13 click13 the13 Load13 Saved13 Dashboard13 icon13

Page of 33 71 Page of 33 71

13

18 Select13 the13 saved13 dashboard13 from13 the13 list13

13

19 The13 dashboard13 runs13

13

Page of 34 71 Page of 34 71

Page of 35 71 Page of 35 71

Aler0ng13 with13 ELK13 ELK13 examines13 the13 log13 records13 in13 process13 to13 send13 alerts13 to13 third-shy‐party13 alerng13 event13 management13 and13 social13 media13 applicaons13 As13 shown13 previously13 in13 the13 ELK13 architecture13 the13 ELK13 strategy13 for13 alerng13 is13 to13 send13 alerts13 from13 the13 Logstash13 component13 directly13 Therefore13 log13 records13 and13 alerts13 are13 sent13 to13 both13 Elascsearch13 and13 third-shy‐party13 tools13 at13 the13 same13 instant13 using13 the13 output13 secon13 of13 the13 Logstash13 configuraon13 Any13 alerts13 are13 generated13 during13 the13 filter13 secon13 processing13 of13 the13 Logstash13 configuraon13

Create13 alerts13 in13 Logstash13 The13 Logstash13 filter13 secon13 is13 used13 to13 trigger13 alerts13 based13 on13 the13 log13 records13 being13 processed13 For13 example13 log13 records13 can13 be13 checked13 for13 errors13 warnings13 crashed13 or13 stopped13 messages13 and13 generate13 an13 alert13

Here13 is13 an13 example13 Logstash13 filter13 code13

13 if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

The13 specific13 tags13 are13 set13 against13 the13 log13 records13 to13 trigger13 the13 alert13 later13 in13 the13 configuraon13 in13 the13 output13 secon13

Control13 the13 number13 of13 generated13 alerts13 using13 the13 Logstash13 throcle13 plugin13 so13 that13 either13 a13 lot13 of13 instances13 in13 a13 fixed13 me13 period13 are13 required13 before13 an13 alert13 is13 sent13 or13 a13 specific13 number13 of13 instances13 triggers13 the13 alert13

Page of 36 71 Page of 36 71

Edit13 the13 plugin13 parameters13 to13 meet13 system13 requirements13

13 13 13 13 13 if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 10

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle13

after_count =gt 20

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 30

period =gt 1200

key =gt servicename_alarm_warning

Page of 37 71 Page of 37 71

add_tag =gt servicename_throttled

13 13 13 13 13 13 13 13 13

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 40

period =gt 1500

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

Enrich13 alerts13 with13 message13 detail13 Some13 of13 the13 third-shy‐party13 integraons13 benefit13 from13 having13 components13 of13 the13 raw13 log13 record13 sent13 in13 addion13 to13 the13 raw13 record13 or13 fields13 in13 the13 record13 edited13 transformed13 or13 translated13

Grok13 Logstash13 allows13 the13 log13 record13 to13 be13 broken13 up13 into13 logical13 secons13 and13 each13 secon13 stored13 in13 a13 new13 parameter13 using13 the13 grok13 parameter13 in13 the13 filter13 secon13 Here13 is13 an13 example13 grok13

13 grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

To13 build13 the13 grok13 statement13 Logstash13 offers13 an13 online13 tool13 at13 hcpsgrokdebugherokuappcom13

Translate13 If13 specific13 fields13 in13 the13 log13 record13 are13 coded13 (for13 example13 the13 Bluemix13 app13 name13 is13 oaen13 not13 sent13 in13 the13 syslog13 messages13 instead13 the13 GUID13 is13 sent13 which13 is13 not13 useful)13 then13 the13 filter13 secon13 translate13 plugin13 can13 be13 used13 to13 add13 new13 fields13 by13 holding13 the13 decoded13 versions13 of13 those13 values13

13 translate

field =gt data1

destination =gt data1

override =gt true

Page of 38 71 Page of 38 71

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

dc50f6e4-c052-436a-bf27-95ba392c4228 monverifyservicemanagement

0c677e82-5bfc-4e38-9155-e416249e3ee6 ms-catalog-postpyretic-pomiculture

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

However13 the13 translate13 plugin13 is13 not13 included13 with13 the13 Logstash13 installaon13 In13 the13 next13 secon13 I13 will13 13 describe13 how13 to13 add13 new13 plugins13 to13 the13 Logstash13 instance13

Add13 custom13 plugins13 to13 Logstash13 As13 an13 open13 source13 tool13 Logstash13 encourages13 users13 to13 develop13 content13 for13 all13 users13 to13 share13 Some13 of13 the13 plugins13 that13 were13 created13 by13 our13 community13 include13 a13 translaon13 plugin13 an13 output13 Slack13 plugin13 and13 an13 output13 syslog13 plugin13 Letrsquos13 look13 at13 how13 to13 add13 these13 plugins13 to13 your13 environment13

Filter13 translate13 plugin13 To13 add13 the13 translate13 plugin13

1 Download13 the13 plugin13

2 Copy13 the13 plugin13 file13 logstash-shy‐filter-shy‐translate-shy‐214gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-filter-translate 214

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Output13 Slack13 plugin13 To13 add13 the13 Slack13 plugin13

Page of 39 71 Page of 39 71

1 Download13 the13 plugin13 13

2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐slack-shy‐014gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-output-slack 014

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Output13 syslog13 plugin13 To13 add13 the13 syslog13 plugin13

1 Download13 the13 plugin13 13

2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐syslog-shy‐214gem13 to13 the13 ELK13 Virtual13 Server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-output-syslog 214

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Integrate13 Logstash13 with13 Slack13 To13 send13 generated13 alerts13 to13 Slack13 complete13 these13 steps13

1 Create13 a13 Slack13 integraon13 13

2 Use13 the13 Slack13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 Slack13 integraon13

113 Login13 to13 Slack13 and13 go13 to13 the13 Apps13 page13 13

Page of 40 71 Page of 40 71

13

313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13

13

A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13

13

Page of 42 71 Page of 42 71

The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13

13

The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13

13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

slack

url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5

channel =gt cloud-operations-mon

username =gt nick_cawood

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13

Page of 43 71 Page of 43 71

Page of 44 71 Page of 44 71

In13 Slack13 the13 alerts13 look13 like13 this13

13

Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13

1 Create13 a13 PagerDuty13 integraon13 13

2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13

1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13

13

2 Click13 Add13 New13 Service13

13

313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13

Page of 45 71 Page of 45 71

13

413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13

513 Click13 Add13 Service13

13

The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13

Page of 46 71 Page of 46 71

13

The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt be7eb1ea9c224018923e818d3e891d0b

incident_key =gt logstashhosttype

Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13

Page of 47 71 Page of 47 71

13

Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13

1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13

2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13

3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13

4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13

13 13 http

url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo

http_method =gt post

format =gt json

For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13

13 13 syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

Page of 48 71 Page of 48 71

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13

13

The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13

if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2

This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13

Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13

Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13

Logstash13 severity Shown13 in13 Omnibus13 as

alert Indeterminate

crical Warning

debug 7

emergency Clear

error Minor

informaonal 6

noce Crical

warning Major

Page of 49 71 Page of 49 71

13

Page of 50 71 Page of 50 71

Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13

input

tcp

port =gt 5000

type =gt syslog

filter

mutate

add_field =gt format =gt cf

grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

translate

field =gt data1

destination =gt data1

override =gt true

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

=== Check Alarm Criteria ===

Standard Key Words

if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

=== Alarm Exclusions ===

if [message] =~ (i)sometexthere

mutate remove_tag =gt servicename_alarm

Throttle alarms so we do not hit PD rate limits

Page of 52 71 Page of 52 71

if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 5

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle

after_count =gt 5

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 5

period =gt 1200

key =gt servicename_alarm_warning

add_tag =gt servicename_throttled

Page of 53 71 Page of 53 71

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

output

elasticsearch

hosts =gt localhost

stdout codec =gt rubydebug

if servicename_alarm_err in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

Page of 54 71 Page of 54 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 55 71 Page of 55 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

Page of 56 71 Page of 56 71

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert err message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_error in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

Page of 57 71 Page of 57 71

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 58 71 Page of 58 71

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 59 71 Page of 59 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert error message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

Page of 60 71 Page of 60 71

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

Page of 61 71 Page of 61 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 62 71 Page of 62 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt x

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert fatal message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

Page of 63 71 Page of 63 71

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

Page of 64 71 Page of 64 71

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 65 71 Page of 65 71

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt heavy_exclamation_mark

Page of 66 71 Page of 66 71

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert WARNING message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 67 71 Page of 67 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

Page of 68 71 Page of 68 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 69 71 Page of 69 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

Page of 70 71 Page of 70 71

Page of 71 71 Page of 71 71

Page 4: How to integrate the Elasticsearch Logstash Kibana (ELK ... · How to integrate the Elasticsearch Logstash Kibana (ELK) log analytics stack into IBM Bluemix Nick Cawood IBM Cloud

secons13 a13 variety13 of13 methods13 are13 available13 to13 process13 the13 data13 sent13 from13 the13 filter13 secon13 For13 example13 the13 records13 can13 be13 sent13 to13 external13 alerng13 tools13 such13 as13 PagerDuty13 social13 media13 tools13 such13 as13 Slack13 or13 event13 management13 hubs13 like13 IBM13 Netcool13 Omnibus13 via13 output13 methods13 syslog13 or13 hcp13 13

As13 an13 open13 source13 tool13 non-shy‐Logstash13 employees13 develop13 other13 methods13 and13 share13 them13 with13 the13 user13 community13 for13 addion13 to13 the13 library13 of13 methods13 Logstash13 can13 execute13 (examples13 of13 this13 process13 are13 included13 later13 in13 this13 document)13

Elas0csearch13 Elascsearch13 is13 the13 index13 store13 and13 query13 applicaon13 on13 the13 ELK13 stack13 It13 provides13 a13 standard13 full13 text13 search13 facility13 of13 the13 log13 file13 records13 processed13 by13 Logstash13 The13 ELK13 online13 help13 documentaon13 describes13 Elascsearch13 as13 follows13 ldquoIt13 is13 generally13 used13 as13 the13 underlying13 enginetechnology13 that13 powers13 applicaons13 that13 have13 complex13 search13 features13 and13 requirementsrdquo13

Kibana13 Kibana13 is13 the13 user13 interface13 for13 the13 ELK13 stack13 Kibana13 integrates13 with13 Elascsearch13 to13 visualise13 the13 results13 of13 searches13 and13 allow13 the13 creaon13 of13 charts13 and13 dashboards13 based13 on13 the13 searches13 The13 searches13 charts13 and13 dashboard13 can13 visualise13 real-shy‐me13 or13 historical13 log13 records13 Users13 access13 Kibana13 through13 an13 hcp13 URL13 from13 a13 client13 web13 browser13

ELK13 System13 Architecture13 The13 ELK13 stack13 can13 be13 described13 as13 follows13

Page of 4 71 Page of 4 71

13

Page of 5 71 Page of 5 71

Service13 Management13 architecture13 The13 IBM13 Cloud13 Architecture13 Center13 created13 a13 standard13 reference13 architecture13 for13 how13 to13 set13 up13 your13 cloud13 environemnt13 to13 handle13 incident13 management13 13

View13 the13 standard13 Service13 Management13 architecure13

13 13

Log13 Monitoring13 System13 architecture13 The13 following13 diagram13 shows13 a13 standard13 architecture13 for13 the13 log-shy‐monitoring13 system13 described13 in13 this13 tutorial13 13

13

Bluemix13 environment13 system13 architecture13 The13 ELK13 stack13 can13 be13 installed13 into13 a13 variety13 of13 environments13 When13 using13 Bluemix13 the13 following13 figure13 shows13 what13 the13 environment13 might13 look13 like13

Page of 6 71 Page of 6 71

13

The13 Bluemix13 cloud13 plaRorm13 offers13 a13 Public13 network13 and13 access13 to13 US13 South13 and13 UK13 environments13 From13 all13 these13 environments13 third-shy‐party13 cloud-shy‐based13 soluons13 are13 also13 accessible13

The13 ELK13 stack13 is13 installed13 in13 the13 Bluemix13 United13 Kingdom13 environment13 because13 this13 is13 the13 environment13 where13 Bluemix13 has13 virtual13 servers13 available13 for13 provisioning13 Other13 applicaons13 in13 the13 CASE13 project13 were13 also13 installed13 on13 virtual13 servers13 in13 the13 Bluemix13 United13 Kingdom13 region13

Bluemix13 apps13 were13 used13 to13 create13 a13 system13 to13 be13 monitored13 by13 ELK13 These13 apps13 were13 hosted13 in13 the13 Bluemix13 US13 South13 environment13 using13 Bluemix13 microservices13 which13 produce13 logs13 during13 normal13 operaon13 and13 during13 crashes13 and13 outages13

Through13 the13 Bluemix13 Cloud13 Foundry13 process13 the13 log13 records13 can13 be13 broadcast13 from13 the13 Bluemix13 apps13 so13 Logstash13 can13 receive13 them13 and13 send13 them13 to13 Elascsearch13 within13 the13 ELK13 stack13 or13 other13 virtual13 servers13 (hosng13 other13 IBM13 or13 third-shy‐party13 products)13 in13 the13 Bluemix13 United13 Kingdom13 environment13 or13 to13 third-shy‐party13 applicaons13 in13 the13 public13 cloud13 (Pager13 Duty13 Slack13 etc)13

Crea0ng13 a13 virtual13 server13 in13 Bluemix13 With13 Bluemix13 you13 can13 provision13 a13 virtual13 server13 in13 the13 cloud13 to13 install13 standalone13 or13 integrated13 system13 components13 Not13 all13 Bluemix13 regions13 offer13 this13 service13 all13 the13 me13 due13 to13 beta-shy‐project13 constraints13 In13 this13 arclersquos13 examples13 virtual13 servers13 were13 allowed13 to13 be13 provisioned13 in13 the13 United13 Kingdom13 region13 For13 more13 informaon13 read13 how13 to13 provision13 a13 Bluemix13 virtual13 server13 including13 some13 simple13 setup13 steps13 for13 ease13 of13 use13

Page of 7 71 Page of 7 71

Create13 an13 SSH13 key13 To13 access13 your13 virtual13 server13 once13 it13 is13 provisioned13 allocate13 an13 SSH13 key13 to13 the13 virtual13 server13

To13 generate13 a13 publicprivate13 RSA13 key13 pair13 follow13 these13 steps13

1 In13 a13 UNIX13 or13 Linuxreg13 window13 run13 the13 following13 command13 specifying13 a13 ltuniquekeynamegt13 of13 your13 your13 choice13 and13 a13 password13 or13 passphrase13 (memorize13 the13 password)$13 ssh-shy‐keygen13 -shy‐t13 rsa13 -shy‐f13 ltuniquekeynamegt13

2 Enter13 the13 passphrase13 (empty13 for13 no13 passphrase)13

3 Enter13 the13 same13 passphrase13 again13

Your13 idenficaon13 has13 been13 saved13 in13 ltuniquekeynamegt13

Your13 public13 key13 has13 been13 saved13 in13 ltuniquekeynamegtpub13

The13 key13 fingerprint13 is13

d6268464506525930ce4632c693c203c13 ltusergtltservergt13

The13 keys13 randomart13 image13 is13

+--[ RSA 2048]----+

|o ++=o |

| E o ooo |

| |

| + o |

| S o |

| o |

| |

| |

| |

+-----------------+

Two13 keys13 are13 created13 public13 and13 private13 The13 public13 key13 contents13 are13 used13 to13 create13 a13 virtual13 server13 with13 a13 public13 IP13 address13 The13 private13 key13 is13 used13 to13 access13 the13 virtual13 server13 from13 a13 client13 machine13

Page of 8 71 Page of 8 71

Provisioning13 a13 virtual13 server13 1 A13 Bluemix13 account13 is13 required13 for13 this13 process13 Sign13 up13 here13 and13 log13 in13 Make13 sure13 your13

environment13 is13 correct13 For13 example13 if13 you13 are13 in13 the13 United13 Kingdom13 select13 the13 United13 Kingdom13 region13 in13 the13 Bluemix13 console13

13

13

2 Add13 a13 virtual13 server13 to13 your13 Bluemix13 environment13 Click13 Run13 Virtual13 Servers13

13

3 Select13 an13 OS13 image13 from13 the13 EXISTING13 list13 For13 ELK13 select13 Centos13 OS13 (use13 latest13 available13 version)13

Page of 9 71 Page of 9 71

4 Name13 the13 virtual13 server13

5 Make13 sure13 Assign13 public13 IP13 address13 is13 checked13

13

6 Click13 Add13 Key13 to13 add13 the13 SSH13 key13 you13 created13 previously13 Copy13 the13 contents13 of13 the13 public13 key13 into13 the13 Public13 Key13 to13 import13 field13

Page of 10 71 Page of 10 71

13

7 Click13 OK13 Then13 click13 CREATE13

The13 virtual13 server13 is13 provisioned13 and13 added13 to13 the13 Bluemix13 dashboard13 under13 VIRTUAL13 SERVERS13

Page of 11 71 Page of 11 71

13

Set13 up13 the13 virtual13 server13

Log13 on13 to13 the13 virtual13 server13 Now13 you13 can13 log13 on13 to13 the13 virtual13 server13 and13 add13 new13 users13 and13 programs13 to13 make13 it13 easier13 to13 access13 the13 virtual13 server13

To13 access13 the13 server13 go13 to13 the13 UNIX13 or13 Linux13 window13 that13 you13 used13 to13 create13 the13 SSH13 key13 In13 the13 directory13 where13 the13 ltuniquekeynamegt13 files13 exist13 run13 the13 following13 command13 referencing13 the13 SSH13 keyname13 the13 public13 IP13 address13 Bluemix13 allocated13 for13 the13 virtual13 server13 and13 the13 password13 for13 the13 SSH13 key13

[userserver ~]$ ssh -i ltuniquekeynamegt ibmcloudltpublicIPaddressgt

Enter13 the13 passphrase13 for13 the13 key13 ltuniquekeynamegt13

Last login Tue Jul 5 125546 2016 from 108-210-96-231lightspeedrlghncsbcglobalnet

[ibmcloudelkforcase ~]$

ibmcloud13 is13 the13 default13 user13 for13 the13 Bluemix13 virtual13 servers13 and13 allows13 SSH13 to13 access13 the13 server13

Acces13 the13 root13 user13 To13 access13 the13 root13 user13 log13 on13 as13 ibmcloud13 user13 and13 run13 the13 following13 command13

[ibmcloudelkforcase ~]$ sudo bash

[rootelkforcase ibmcloud]

The13 root13 user13 can13 be13 used13 to13 add13 programs13 to13 the13 server13 and13 create13 and13 edit13 users13 as13 required13 to13 access13 applicaons13 installed13 on13 the13 virtual13 server13

Page of 12 71 Page of 12 71

Copy13 files13 to13 the13 virtual13 server13 To13 move13 installaon13 files13 to13 the13 server13 go13 to13 the13 UNIX13 or13 Linux13 window13 you13 used13 to13 create13 the13 SSH13 key13 and13 the13 directory13 where13 the13 ltuniquekeynamegt13 files13 exist13 Move13 the13 installaon13 files13 to13 the13 same13 locaon13

Run13 the13 following13 command13 referencing13 the13 SSH13 keyname13 the13 public13 IP13 address13 Bluemix13 allocated13 for13 the13 virtual13 server13 and13 the13 password13 for13 the13 SSH13 key13

[userserver ~]$ scp ndashi ltuniquekeynamegt ltINSTALLATIONFILENAMEgt ibmcloudltpublicIPaddressgthomeibmcloud

Enter13 the13 passphrase13 for13 key13 cloudkey313

This13 copies13 the13 file13 using13 the13 ibmcloud13 user13 to13 the13 homeibmcloud13 directory13 on13 the13 virtual13 server13

Install13 and13 access13 the13 virtual13 server13 via13 VNC13 server13 Using13 the13 ssh13 command13 to13 access13 the13 virtual13 server13 is13 not13 the13 most13 stable13 method13 and13 does13 not13 offer13 X-shy‐Windows13 support13 Therefore13 we13 suggest13 that13 you13 use13 a13 VNC13 client13 to13 access13 the13 virtual13 server13

Installing13 a13 VNC13 client13 should13 be13 done13 as13 the13 root13 user13 Use13 the13 yum13 command13 as13 follows13

[ibmcloudelkforcase ~]$ yum groupinstall GNOME Desktop

[ibmcloudelkforcase ~]$ yum install tigervnc-server

These13 commands13 first13 install13 a13 desktop13 style13 OS13 user13 interface13 and13 then13 a13 VNC13 client13 to13 remotely13 access13 the13 desktop13 style13 OS13

Then13 as13 either13 the13 root13 user13 or13 another13 user13 (example13 below13 uses13 elk)13 run13 the13 following13 command13 to13 create13 the13 VNC13 client13 connecon13

[elkelkforcase ~]$ vncserver

New elkforcase2 (elk) desktop is elkforcase2

Starting applications specified in homeelkvncxstartup

Log file is homeelkvncelkforcase2log

[elkelkforcase ~]$

Install13 a13 VNC13 client13 on13 the13 client13 machine13 used13 to13 access13 the13 Internet13 and13 (in13 the13 example13 above)13 use13 the13 command13 ltpublicIPadddressgt213 and13 the13 password13 for13 ELK13 to13 access13 the13 virtual13 server13

Using13 a13 VNC13 client13 means13 that13 you13 can13 leave13 long13 duraon13 installaons13 running13 without13 fear13 of13 the13 terminal13 losing13 connecon13 and13 the13 installaons13 failing13

Install13 the13 ELK13 stack13 Download13 the13 source13 soaware13 code13 for13 the13 ELK13 stack13 from13 the13 ELK13 website13 13

At13 the13 me13 of13 wring13 the13 downloaded13 soaware13 files13 were13

Page of 13 71 Page of 13 71

13

Transfer13 the13 files13 to13 the13 virtual13 server13 (using13 the13 scp13 method13 detailed13 above)13

Install13 Elas0csearch13 As13 a13 root13 user13 create13 a13 non-shy‐root13 user13 with13 the13 adduser13 command13

Create13 a13 directory13 to13 install13 Elascsearch13 into13 and13 untar13 the13 contents13 of13 the13 Elascsearch13 install13 file13 into13 that13 directory13

[elkelkforcase]$ cd home

[elkelkforcase]$ mkdir elasticsearch

[elkelkforcase]$ cp homeibmcloudelasticsearch-233targz homeelasticsearch

[elkelkforcase]$ cd elasticsearch

[elkelkforcase]$ gunzip elasticsearch-233targz

[elkelkforcase]$ untar elasticsearch-233tar

Elascsearch13 is13 now13 installed13

Start13 Elas0csearch13 To13 start13 Elascsearch13 go13 to13 the13 bin13 directory13 as13 the13 non-shy‐root13 user13 when13 you13 installed13 it13 and13 run13 the13 elascsearch13 executable13

[elkelkforcase]$ cd homeelasticsearchelasticsearch-233bin

[elkelkforcase]$ elasticsearch amp

[2] 22211

[elkelkforcase]$ [2016-07-05 133857730][INFO ][node] [Arsenal] version[233] pid[22211] build[218bdf12016-05-17T154004Z]

[2016-07-05 133857740][INFO ][node] [Arsenal] initializing

[2016-07-05 133858984][INFO ][plugins] [Arsenal] modules [reindex lang-expression lang-groovy] plugins [] sites []

[2016-07-05 133859146][INFO ][env] [Arsenal] using [1] data paths mounts [[ (rootfs)]] net usable_space [155gb] net total_space [199gb] spins [unknown] types [rootfs]

[2016-07-05 133859146][INFO ][env] [Arsenal] heap size [10156mb] compressed ordinary object pointers [true]

Page of 14 71 Page of 14 71

[2016-07-05 133859147][WARN ][env] [Arsenal] max file descriptors [4096] for elasticsearch process likely too low consider increasing to at least [65536]

[2016-07-05 133903526][INFO ][node] [Arsenal] initialized

[2016-07-05 133903526][INFO ][node] [Arsenal] starting

To13 test13 that13 Elascsearch13 is13 running13 correctly13 open13 the13 browser13 in13 the13 VNC13 client13 and13 run13 localhost920013

13

Install13 Kibana13 ELK13 recommends13 installing13 and13 running13 Kibana13 as13 root13 13

1 As13 root13 user13 create13 an13 installaon13 directory13 and13 untar13 the13 installaon13 file13

[rootelkforcase]$ cd home

[rootelkforcase]$ mkdir kibana

[rootelkforcase]$ cp homeibmcloudkibana-451-linux-x64targz homekibana

[rootelkforcase]$ cd kibana

[rootelkforcase]$ gunzip kibana-451-linux-x64targz

[rootelkforcase]$ untar kibana-451-linux-x64tar

2 Modify13 the13 manifestyml13 configuraon13 file13 Go13 to13 the13 kibana-shy‐451-shy‐linux-shy‐x6413 directory13

[rootelkforcase]$ cd homekibanakibana-451-linux-x64

3 Edit13 the13 manifestyml13 file13 so13 the13 contents13 are13

applicaons13

-shy‐ name13 kibana-shy‐in-shy‐bluemix13

memory13 128M13

host13 kibana-shy‐in-shy‐bluemix13

buildpack13 hcpsgithubcomcloudfoundry-shy‐communitynginx-shy‐buildpackgit13

Page of 15 71 Page of 15 71

4 Edit13 the13 configkibanayml13 file13 so13 the13 contents13 are13

Kibana is served by a back end server This controls which port to use

port 5601

The host to bind the server to

host 0000

If you are running kibana behind a proxy and want to mount it at a path

specify that path here The basePath cant end in a slash

serverbasePath

The maximum payload size in bytes on incoming server requests

servermaxPayloadBytes 1048576

The Elasticsearch instance to use for all your queries

elasticsearchurl httplocalhost9200

preserve_elasticsearch_host true will send the hostname specified in `elasticsearch` If you set it to false

then the host you use to connect to this Kibana instance will be sent

elasticsearchpreserveHost true

Kibana uses an index in Elasticsearch to store saved searches visualizations

and dashboards It will create a new index if it doesnt already exist

kibanaindex kibana

The default application to load

kibanadefaultAppId discover

If your Elasticsearch is protected with basic auth these are the user credentials

used by the Kibana server to perform maintenance on the kibana_index at startup Your

Kibana users will still need to authenticate with Elasticsearch (which is proxied through

the Kibana server)

Page of 16 71 Page of 16 71

elasticsearchusername user

elasticsearchpassword pass

SSL for outgoing requests from the Kibana Server to the browser (PEM formatted)

serversslcert pathtoyourservercrt

serversslkey pathtoyourserverkey

Optional setting to validate that your Elasticsearch backend uses the same key files (PEM formatted)

elasticsearchsslcert pathtoyourclientcrt

elasticsearchsslkey pathtoyourclientkey

If you need to provide a CA certificate for your Elasticsearch instance put

the path of the pem file here

elasticsearchsslca pathtoyourCApem

Set to false to have a complete disregard for the validity of the SSL

certificate

elasticsearchsslverify true

Time in milliseconds to wait for elasticsearch to respond to pings defaults to

request_timeout setting

elasticsearchpingTimeout 1500

Time in milliseconds to wait for responses from the back end or elasticsearch

This must be gt 0

elasticsearchrequestTimeout 30000

Time in milliseconds for Elasticsearch to wait for responses from shards

Set to 0 to disable

elasticsearchshardTimeout 0

Time in milliseconds to wait for Elasticsearch at Kibana startup before retrying

elasticsearchstartupTimeout 5000

elasticsearchstartupTimeout 30000

Page of 17 71 Page of 17 71

Set the path to where you would like the process id file to be created

pidfile varrunkibanapid

If you would like to send the log output to a file you can set the path below

loggingdest stdout

Set this to true to suppress all logging output

loggingsilent false

Set this to true to suppress all logging output except for error messages

loggingquiet false

Set this to true to log all events including system usage information and all requests

loggingverbose false

Kibana13 is13 now13 installed13

Start13 Kibana13 To13 start13 Kibana13 go13 to13 the13 bin13 directory13 and13 run13 the13 kibana13 executable13

[rootelkforcase]$

[rootelkforcase]$ cd homekibanakibana-451-linux-x64bin

[rootelkforcase]$ kibana amp

[1] 22283

[rootelkforcase] log [095900542] [info][status][pluginkibana] Status changed from uninitialized to green - Ready

log [095900600] [info][status][pluginelasticsearch] Status changed from uninitialized to yellow - Waiting for Elasticsearch

log [095900689] [info][status][pluginkbn_vislib_vis_types] Status changed from uninitialized to green - Ready

log [095900717] [info][status][pluginmarkdown_vis] Status changed from uninitialized to green - Ready

log [095900721] [info][status][pluginmetric_vis] Status changed from uninitialized to green - Ready

log [095900814] [info][status][pluginspyModes] Status changed from uninitialized to green - Ready

log [095900934] [info][status][pluginstatusPage] Status changed from uninitialized to green - Ready

Page of 18 71 Page of 18 71

log [095900941] [info][status][plugintable_vis] Status changed from uninitialized to green - Ready

log [095901067] [info][listening] Server running at http00005601

log [095906083] [info][status][pluginelasticsearch] Status changed from yellow to yellow - No existing Kibana index found

log [095908942] [info][status][pluginelasticsearch] Status changed from yellow to green - Kibana index ready

Kibana13 is13 now13 started13

Broadcast13 Bluemix13 app13 log13 records13 Bluemix13 apps13 (which13 are13 shown13 in13 the13 US13 South13 Bluemix13 region13 of13 the13 System13 Architecture13 diagram)13 will13 produce13 logs13 in13 a13 syslog13 rfc542413 format13 using13 a13 Bluemix13 tool13 called13 Loggregator13

For13 Logstash13 to13 receive13 these13 logs13 each13 Bluemix13 app13 needs13 to13 broadcast13 these13 logs13 To13 do13 that13 run13 the13 following13 commands13 on13 each13 Bluemix13 app13 using13 the13 public13 IP13 address13 of13 the13 ELK13 virtual13 server13 the13 ELK13 tag13 to13 trace13 the13 process13 and13 insert13 the13 name13 of13 each13 app13 in13 turn13

cf cups elk -l syslogltpublicIPaddress5000 cf bind-service ltmyappgt elk

cf restage ltmyappgt

Install13 Logstash13 To13 install13 Logstash13 as13 root13 user13 create13 a13 directory13 for13 the13 installaon13 and13 untar13 the13 installaon13 file13 there13

13 13 [rootelkforcase]$ cd home

[rootelkforcase]$ mkdir logstash

[rootelkforcase]$ cp homeibmcloudlogstash-233targz homelogstash

[rootelkforcase]$ cd logstash

[rootelkforcase]$ gunzip logstash-233targz

[rootelkforcase]$ untar logstash-233tar

Write13 Logstash13 configura0on13 Logstash13 requires13 a13 configuraon13 file13 to13 be13 wricen13 to13 receive13 transform13 and13 output13 the13 log13 records13 to13 Elascsearch13 This13 secon13 explains13 the13 basic13 requirements13 of13 processing13 Bluemix13 app13 log13 records13 into13 Elascsearch13 (more13 complex13 configuraons13 are13 detailed13 later13 in13 this13 document13 for13 the13 integraons13 with13 third-shy‐party13 tools)13

Create13 a13 conf13 file13 as13 root13 user13

[rootelkforcase]$ cd homelogstashlogstash-233

[rootelkforcase]$ mkdir config

Page of 19 71 Page of 19 71

[rootelkforcase]$ cd config

[rootelkforcase]$ touch loggregatorconf

Edit13 the13 loggregator13 file13 with13 the13 following13 contents13

13 Input13 secon13 defines13 which13 log13 records13 to13 receive13

13 port13 and13 type13 reflect13 method13 selected13 for13 Cloud13 Foundry13 to13 broadcast13 the13 records13

input

tcp

port =gt 5000

type =gt syslog

13 Filter13 secon13 ndash13 transforms13 the13 data13

filter

Add a field to identify the records as Cloud Foundry based

mutate

add_field =gt format =gt cf

13 Output13 secon13 ndash13 sends13 the13 data13 to13 its13 desnaon(s)13

output

13 Use13 the13 elascsearch13 plugin13 to13 send13 the13 data13 to13 the13 elascsearch13 instance13

13 elasticsearch

hosts =gt localhost

stdout codec =gt rubydebug

13 End13 of13 Config13

Save13 the13 file13 Verify13 the13 format13 of13 the13 file13 is13 acceptable13 by13 running13 the13 following13 commands13

[rootelkforcase]$ cd homelogstashlogstash-233

[rootelkforcase]$ binlogstash agent --configtest -f configloggregatorconf

Page of 20 71 Page of 20 71

Configuraon13 is13 OK13

Start13 Logstash13 Aaer13 you13 have13 wricen13 and13 verified13 the13 Logstash13 configuraon13 you13 can13 start13 Logstash13 to13 process13 records13 from13 the13 Bluemix13 apps13 (via13 Cloud13 Foundry)13 Run13 the13 following13 as13 root13 user13

[rootelkforcase]$ cd homelogstashlogstash-233

[rootelkforcase]$ binlogstash agent -f configloggregatorconf

Logstash13 is13 now13 started13

Monitor13 Logstash13 processing13 Once13 Logstash13 is13 running13 and13 log13 records13 are13 being13 produced13 by13 the13 Bluemix13 apps13 you13 can13 monitor13 progress13 in13 the13 same13 window13 that13 was13 used13 to13 start13 Logstash13 Output13 is13 similar13 to13 the13 following13

13

The13 log13 records13 being13 broadcast13 from13 the13 Bluemix13 apps13 are13 now13 sent13 to13 Elascsearch13 from13 the13 Logstash13 output13 configuraon13 Records13 are13 sent13 in13 real13 me13 If13 Logstash13 is13 not13 running13 while13 log13 records13 are13 being13 produced13 then13 these13 records13 will13 not13 be13 processed13 by13 Logstash13 and13 will13 not13 be13 stored13 in13 Elascsearch13

Page of 21 71 Page of 21 71

Use13 ELK13 The13 log13 records13 from13 the13 Bluemix13 apps13 are13 now13 being13 stored13 and13 indexed13 by13 the13 Elascsearch13 component13 This13 data13 can13 be13 accessed13 via13 the13 Kibana13 applicaon13

Kibana13 Discover13 page13 Using13 a13 web13 browser13 either13 directly13 from13 a13 client13 or13 using13 the13 VNC13 client13 access13 Kibana13 using13 the13 publicIPaddress13 of13 the13 ELK13 virtual13 server13

hcpltpublicIPaddressgt560113

No13 userpassword13 credenals13 are13 required13

Kibana13 defaults13 to13 a13 search13 of13 all13 records13 ()13 in13 the13 last13 1513 minutes13 and13 returns13 the13 results13

13

More13 complex13 searches13 are13 possible13 by13 changing13 the13 me13 filter13 and13 supplying13 search13 queries13

Change13 the13 0me13 filter13 To13 change13 the13 me13 ranges13 click13 the13 clock13 symbol13 in13 the13 top-shy‐right13 of13 the13 Kibana13 Discover13 page13

13

You13 will13 see13 the13 following13 opons13

13

Page of 22 71 Page of 22 71

A13 variety13 of13 quick13 me13 ranges13 can13 be13 selected13 or13 you13 can13 switch13 to13 relave13 or13 absolute13 me13 ranges13

Quick13 me13 ranges13 allow13 switching13 between13 last13 X13 minutes13 to13 last13 Y13 hours13 or13 Z13 days13 etc13

Relave13 me13 ranges13 allow13 more13 specific13 versions13 of13 the13 Quick13 opons13 from13 specific13 minuteshoursdays13 to13 now13

13

Absolute13 me13 ranges13 specify13 the13 start13 and13 end13 mes13 for13 the13 searches13

13

Custom13 search13 queries13 You13 can13 use13 the13 text13 window13 of13 the13 Discover13 page13 to13 search13 for13 matches13 within13 the13 log13 records13 You13 can13 use13 Apache13 Lucene13 or13 Elascsearch13 query13 DSL13 syntax13

13

Page of 23 71 Page of 23 71

Saved13 searches13 You13 can13 save13 any13 custom13 search13 syntax13 and13 choice13 of13 me13 range13 for13 future13 ease13 of13 use13 Any13 relave13 or13 absolute13 me13 frame13 is13 saved13 as13 part13 of13 the13 search13 13

To13 save13 a13 search13

1 Set13 up13 and13 run13 the13 required13 syntax13 and13 me13 frame13

13

2 Click13 the13 Save13 Search13 icon13

13

3 Type13 in13 a13 name13 for13 the13 saved13 search13

Page of 24 71 Page of 24 71

13

7 To13 delete13 or13 update13 saved13 searches13 click13 13 manage13 saved13 searches13

13

Use13 the13 edit13 icons13 to13 make13 any13 changes13

Page of 26 71 Page of 26 71

13

Build13 simple13 Kibana13 histograms13 Kibana13 allows13 you13 to13 build13 charts13 and13 dashboards13 based13 on13 the13 search13 capability13 offered13 by13 the13 Discover13 page13 Saved13 searches13 can13 be13 used13 as13 the13 basis13 for13 charts13 and13 dashboards13 built13 using13 those13 charts13 To13 build13 a13 simple13 histogram13

1 Select13 the13 Visualize13 page13

13

2 Select13 Ver0cal13 Bar13 Chart13

13

3 To13 use13 a13 saved13 search13 click13 From13 a13 saved13 search13

Page of 27 71 Page of 27 71

13

4 Select13 a13 previously13 saved13 search13

13

5 Select13 to13 edit13 the13 x-shy‐axis

6 Edit13 the13 fields13 with13 informaon13 about13 the13 aggregaon13 field13 interval13 and13 custom13 label13

Page of 28 71 Page of 28 71

13

Page of 29 71 Page of 29 71

7 Click13 the13 green13 arrow13 The13 histogram13 is13 created13 and13 run13 based13 on13 the13 saved13 search

13

8 To13 save13 the13 histogram13 click13 the13 Save13 Visualiza0on13 icon13 and13 add13 a13 tle13

13

13

9 Click13 Save13

10 To13 view13 the13 visualisaon13 click13 the13 Dashboard13 page13

Page of 30 71 Page of 30 71

13

11 Click13 the13 Add13 Visualiza0on13 icon13

13

12 Select13 the13 visiualizaon13 that13 you13 just13 created13

13

Page of 31 71 Page of 31 71

13

13 To13 save13 the13 dashboard13 click13 the13 Save13 Dashboard13 icon13

Page of 32 71 Page of 32 71

13

14 Add13 a13 tle13 and13 select13 whether13 to13 store13 the13 selected13 me13 frame13 with13 the13 dashboard13

13

15 Click13 Save13

16 You13 can13 edit13 the13 shape13 and13 number13 of13 charts13 and13 save13 the13 dashboard13

17 To13 re-shy‐run13 the13 dashboard13 select13 the13 Dashboard13 page13 and13 click13 the13 Load13 Saved13 Dashboard13 icon13

Page of 33 71 Page of 33 71

13

18 Select13 the13 saved13 dashboard13 from13 the13 list13

13

19 The13 dashboard13 runs13

13

Page of 34 71 Page of 34 71

Page of 35 71 Page of 35 71

Aler0ng13 with13 ELK13 ELK13 examines13 the13 log13 records13 in13 process13 to13 send13 alerts13 to13 third-shy‐party13 alerng13 event13 management13 and13 social13 media13 applicaons13 As13 shown13 previously13 in13 the13 ELK13 architecture13 the13 ELK13 strategy13 for13 alerng13 is13 to13 send13 alerts13 from13 the13 Logstash13 component13 directly13 Therefore13 log13 records13 and13 alerts13 are13 sent13 to13 both13 Elascsearch13 and13 third-shy‐party13 tools13 at13 the13 same13 instant13 using13 the13 output13 secon13 of13 the13 Logstash13 configuraon13 Any13 alerts13 are13 generated13 during13 the13 filter13 secon13 processing13 of13 the13 Logstash13 configuraon13

Create13 alerts13 in13 Logstash13 The13 Logstash13 filter13 secon13 is13 used13 to13 trigger13 alerts13 based13 on13 the13 log13 records13 being13 processed13 For13 example13 log13 records13 can13 be13 checked13 for13 errors13 warnings13 crashed13 or13 stopped13 messages13 and13 generate13 an13 alert13

Here13 is13 an13 example13 Logstash13 filter13 code13

13 if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

The13 specific13 tags13 are13 set13 against13 the13 log13 records13 to13 trigger13 the13 alert13 later13 in13 the13 configuraon13 in13 the13 output13 secon13

Control13 the13 number13 of13 generated13 alerts13 using13 the13 Logstash13 throcle13 plugin13 so13 that13 either13 a13 lot13 of13 instances13 in13 a13 fixed13 me13 period13 are13 required13 before13 an13 alert13 is13 sent13 or13 a13 specific13 number13 of13 instances13 triggers13 the13 alert13

Page of 36 71 Page of 36 71

Edit13 the13 plugin13 parameters13 to13 meet13 system13 requirements13

13 13 13 13 13 if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 10

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle13

after_count =gt 20

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 30

period =gt 1200

key =gt servicename_alarm_warning

Page of 37 71 Page of 37 71

add_tag =gt servicename_throttled

13 13 13 13 13 13 13 13 13

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 40

period =gt 1500

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

Enrich13 alerts13 with13 message13 detail13 Some13 of13 the13 third-shy‐party13 integraons13 benefit13 from13 having13 components13 of13 the13 raw13 log13 record13 sent13 in13 addion13 to13 the13 raw13 record13 or13 fields13 in13 the13 record13 edited13 transformed13 or13 translated13

Grok13 Logstash13 allows13 the13 log13 record13 to13 be13 broken13 up13 into13 logical13 secons13 and13 each13 secon13 stored13 in13 a13 new13 parameter13 using13 the13 grok13 parameter13 in13 the13 filter13 secon13 Here13 is13 an13 example13 grok13

13 grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

To13 build13 the13 grok13 statement13 Logstash13 offers13 an13 online13 tool13 at13 hcpsgrokdebugherokuappcom13

Translate13 If13 specific13 fields13 in13 the13 log13 record13 are13 coded13 (for13 example13 the13 Bluemix13 app13 name13 is13 oaen13 not13 sent13 in13 the13 syslog13 messages13 instead13 the13 GUID13 is13 sent13 which13 is13 not13 useful)13 then13 the13 filter13 secon13 translate13 plugin13 can13 be13 used13 to13 add13 new13 fields13 by13 holding13 the13 decoded13 versions13 of13 those13 values13

13 translate

field =gt data1

destination =gt data1

override =gt true

Page of 38 71 Page of 38 71

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

dc50f6e4-c052-436a-bf27-95ba392c4228 monverifyservicemanagement

0c677e82-5bfc-4e38-9155-e416249e3ee6 ms-catalog-postpyretic-pomiculture

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

However13 the13 translate13 plugin13 is13 not13 included13 with13 the13 Logstash13 installaon13 In13 the13 next13 secon13 I13 will13 13 describe13 how13 to13 add13 new13 plugins13 to13 the13 Logstash13 instance13

Add13 custom13 plugins13 to13 Logstash13 As13 an13 open13 source13 tool13 Logstash13 encourages13 users13 to13 develop13 content13 for13 all13 users13 to13 share13 Some13 of13 the13 plugins13 that13 were13 created13 by13 our13 community13 include13 a13 translaon13 plugin13 an13 output13 Slack13 plugin13 and13 an13 output13 syslog13 plugin13 Letrsquos13 look13 at13 how13 to13 add13 these13 plugins13 to13 your13 environment13

Filter13 translate13 plugin13 To13 add13 the13 translate13 plugin13

1 Download13 the13 plugin13

2 Copy13 the13 plugin13 file13 logstash-shy‐filter-shy‐translate-shy‐214gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-filter-translate 214

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Output13 Slack13 plugin13 To13 add13 the13 Slack13 plugin13

Page of 39 71 Page of 39 71

1 Download13 the13 plugin13 13

2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐slack-shy‐014gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-output-slack 014

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Output13 syslog13 plugin13 To13 add13 the13 syslog13 plugin13

1 Download13 the13 plugin13 13

2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐syslog-shy‐214gem13 to13 the13 ELK13 Virtual13 Server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-output-syslog 214

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Integrate13 Logstash13 with13 Slack13 To13 send13 generated13 alerts13 to13 Slack13 complete13 these13 steps13

1 Create13 a13 Slack13 integraon13 13

2 Use13 the13 Slack13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 Slack13 integraon13

113 Login13 to13 Slack13 and13 go13 to13 the13 Apps13 page13 13

Page of 40 71 Page of 40 71

13

313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13

13

A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13

13

Page of 42 71 Page of 42 71

The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13

13

The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13

13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

slack

url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5

channel =gt cloud-operations-mon

username =gt nick_cawood

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13

Page of 43 71 Page of 43 71

Page of 44 71 Page of 44 71

In13 Slack13 the13 alerts13 look13 like13 this13

13

Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13

1 Create13 a13 PagerDuty13 integraon13 13

2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13

1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13

13

2 Click13 Add13 New13 Service13

13

313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13

Page of 45 71 Page of 45 71

13

413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13

513 Click13 Add13 Service13

13

The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13

Page of 46 71 Page of 46 71

13

The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt be7eb1ea9c224018923e818d3e891d0b

incident_key =gt logstashhosttype

Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13

Page of 47 71 Page of 47 71

13

Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13

1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13

2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13

3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13

4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13

13 13 http

url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo

http_method =gt post

format =gt json

For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13

13 13 syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

Page of 48 71 Page of 48 71

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13

13

The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13

if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2

This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13

Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13

Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13

Logstash13 severity Shown13 in13 Omnibus13 as

alert Indeterminate

crical Warning

debug 7

emergency Clear

error Minor

informaonal 6

noce Crical

warning Major

Page of 49 71 Page of 49 71

13

Page of 50 71 Page of 50 71

Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13

input

tcp

port =gt 5000

type =gt syslog

filter

mutate

add_field =gt format =gt cf

grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

translate

field =gt data1

destination =gt data1

override =gt true

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

=== Check Alarm Criteria ===

Standard Key Words

if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

=== Alarm Exclusions ===

if [message] =~ (i)sometexthere

mutate remove_tag =gt servicename_alarm

Throttle alarms so we do not hit PD rate limits

Page of 52 71 Page of 52 71

if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 5

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle

after_count =gt 5

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 5

period =gt 1200

key =gt servicename_alarm_warning

add_tag =gt servicename_throttled

Page of 53 71 Page of 53 71

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

output

elasticsearch

hosts =gt localhost

stdout codec =gt rubydebug

if servicename_alarm_err in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

Page of 54 71 Page of 54 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 55 71 Page of 55 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

Page of 56 71 Page of 56 71

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert err message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_error in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

Page of 57 71 Page of 57 71

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 58 71 Page of 58 71

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 59 71 Page of 59 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert error message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

Page of 60 71 Page of 60 71

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

Page of 61 71 Page of 61 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 62 71 Page of 62 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt x

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert fatal message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

Page of 63 71 Page of 63 71

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

Page of 64 71 Page of 64 71

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 65 71 Page of 65 71

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt heavy_exclamation_mark

Page of 66 71 Page of 66 71

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert WARNING message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 67 71 Page of 67 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

Page of 68 71 Page of 68 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 69 71 Page of 69 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

Page of 70 71 Page of 70 71

Page of 71 71 Page of 71 71

Page 5: How to integrate the Elasticsearch Logstash Kibana (ELK ... · How to integrate the Elasticsearch Logstash Kibana (ELK) log analytics stack into IBM Bluemix Nick Cawood IBM Cloud

13

Page of 5 71 Page of 5 71

Service13 Management13 architecture13 The13 IBM13 Cloud13 Architecture13 Center13 created13 a13 standard13 reference13 architecture13 for13 how13 to13 set13 up13 your13 cloud13 environemnt13 to13 handle13 incident13 management13 13

View13 the13 standard13 Service13 Management13 architecure13

13 13

Log13 Monitoring13 System13 architecture13 The13 following13 diagram13 shows13 a13 standard13 architecture13 for13 the13 log-shy‐monitoring13 system13 described13 in13 this13 tutorial13 13

13

Bluemix13 environment13 system13 architecture13 The13 ELK13 stack13 can13 be13 installed13 into13 a13 variety13 of13 environments13 When13 using13 Bluemix13 the13 following13 figure13 shows13 what13 the13 environment13 might13 look13 like13

Page of 6 71 Page of 6 71

13

The13 Bluemix13 cloud13 plaRorm13 offers13 a13 Public13 network13 and13 access13 to13 US13 South13 and13 UK13 environments13 From13 all13 these13 environments13 third-shy‐party13 cloud-shy‐based13 soluons13 are13 also13 accessible13

The13 ELK13 stack13 is13 installed13 in13 the13 Bluemix13 United13 Kingdom13 environment13 because13 this13 is13 the13 environment13 where13 Bluemix13 has13 virtual13 servers13 available13 for13 provisioning13 Other13 applicaons13 in13 the13 CASE13 project13 were13 also13 installed13 on13 virtual13 servers13 in13 the13 Bluemix13 United13 Kingdom13 region13

Bluemix13 apps13 were13 used13 to13 create13 a13 system13 to13 be13 monitored13 by13 ELK13 These13 apps13 were13 hosted13 in13 the13 Bluemix13 US13 South13 environment13 using13 Bluemix13 microservices13 which13 produce13 logs13 during13 normal13 operaon13 and13 during13 crashes13 and13 outages13

Through13 the13 Bluemix13 Cloud13 Foundry13 process13 the13 log13 records13 can13 be13 broadcast13 from13 the13 Bluemix13 apps13 so13 Logstash13 can13 receive13 them13 and13 send13 them13 to13 Elascsearch13 within13 the13 ELK13 stack13 or13 other13 virtual13 servers13 (hosng13 other13 IBM13 or13 third-shy‐party13 products)13 in13 the13 Bluemix13 United13 Kingdom13 environment13 or13 to13 third-shy‐party13 applicaons13 in13 the13 public13 cloud13 (Pager13 Duty13 Slack13 etc)13

Crea0ng13 a13 virtual13 server13 in13 Bluemix13 With13 Bluemix13 you13 can13 provision13 a13 virtual13 server13 in13 the13 cloud13 to13 install13 standalone13 or13 integrated13 system13 components13 Not13 all13 Bluemix13 regions13 offer13 this13 service13 all13 the13 me13 due13 to13 beta-shy‐project13 constraints13 In13 this13 arclersquos13 examples13 virtual13 servers13 were13 allowed13 to13 be13 provisioned13 in13 the13 United13 Kingdom13 region13 For13 more13 informaon13 read13 how13 to13 provision13 a13 Bluemix13 virtual13 server13 including13 some13 simple13 setup13 steps13 for13 ease13 of13 use13

Page of 7 71 Page of 7 71

Create13 an13 SSH13 key13 To13 access13 your13 virtual13 server13 once13 it13 is13 provisioned13 allocate13 an13 SSH13 key13 to13 the13 virtual13 server13

To13 generate13 a13 publicprivate13 RSA13 key13 pair13 follow13 these13 steps13

1 In13 a13 UNIX13 or13 Linuxreg13 window13 run13 the13 following13 command13 specifying13 a13 ltuniquekeynamegt13 of13 your13 your13 choice13 and13 a13 password13 or13 passphrase13 (memorize13 the13 password)$13 ssh-shy‐keygen13 -shy‐t13 rsa13 -shy‐f13 ltuniquekeynamegt13

2 Enter13 the13 passphrase13 (empty13 for13 no13 passphrase)13

3 Enter13 the13 same13 passphrase13 again13

Your13 idenficaon13 has13 been13 saved13 in13 ltuniquekeynamegt13

Your13 public13 key13 has13 been13 saved13 in13 ltuniquekeynamegtpub13

The13 key13 fingerprint13 is13

d6268464506525930ce4632c693c203c13 ltusergtltservergt13

The13 keys13 randomart13 image13 is13

+--[ RSA 2048]----+

|o ++=o |

| E o ooo |

| |

| + o |

| S o |

| o |

| |

| |

| |

+-----------------+

Two13 keys13 are13 created13 public13 and13 private13 The13 public13 key13 contents13 are13 used13 to13 create13 a13 virtual13 server13 with13 a13 public13 IP13 address13 The13 private13 key13 is13 used13 to13 access13 the13 virtual13 server13 from13 a13 client13 machine13

Page of 8 71 Page of 8 71

Provisioning13 a13 virtual13 server13 1 A13 Bluemix13 account13 is13 required13 for13 this13 process13 Sign13 up13 here13 and13 log13 in13 Make13 sure13 your13

environment13 is13 correct13 For13 example13 if13 you13 are13 in13 the13 United13 Kingdom13 select13 the13 United13 Kingdom13 region13 in13 the13 Bluemix13 console13

13

13

2 Add13 a13 virtual13 server13 to13 your13 Bluemix13 environment13 Click13 Run13 Virtual13 Servers13

13

3 Select13 an13 OS13 image13 from13 the13 EXISTING13 list13 For13 ELK13 select13 Centos13 OS13 (use13 latest13 available13 version)13

Page of 9 71 Page of 9 71

4 Name13 the13 virtual13 server13

5 Make13 sure13 Assign13 public13 IP13 address13 is13 checked13

13

6 Click13 Add13 Key13 to13 add13 the13 SSH13 key13 you13 created13 previously13 Copy13 the13 contents13 of13 the13 public13 key13 into13 the13 Public13 Key13 to13 import13 field13

Page of 10 71 Page of 10 71

13

7 Click13 OK13 Then13 click13 CREATE13

The13 virtual13 server13 is13 provisioned13 and13 added13 to13 the13 Bluemix13 dashboard13 under13 VIRTUAL13 SERVERS13

Page of 11 71 Page of 11 71

13

Set13 up13 the13 virtual13 server13

Log13 on13 to13 the13 virtual13 server13 Now13 you13 can13 log13 on13 to13 the13 virtual13 server13 and13 add13 new13 users13 and13 programs13 to13 make13 it13 easier13 to13 access13 the13 virtual13 server13

To13 access13 the13 server13 go13 to13 the13 UNIX13 or13 Linux13 window13 that13 you13 used13 to13 create13 the13 SSH13 key13 In13 the13 directory13 where13 the13 ltuniquekeynamegt13 files13 exist13 run13 the13 following13 command13 referencing13 the13 SSH13 keyname13 the13 public13 IP13 address13 Bluemix13 allocated13 for13 the13 virtual13 server13 and13 the13 password13 for13 the13 SSH13 key13

[userserver ~]$ ssh -i ltuniquekeynamegt ibmcloudltpublicIPaddressgt

Enter13 the13 passphrase13 for13 the13 key13 ltuniquekeynamegt13

Last login Tue Jul 5 125546 2016 from 108-210-96-231lightspeedrlghncsbcglobalnet

[ibmcloudelkforcase ~]$

ibmcloud13 is13 the13 default13 user13 for13 the13 Bluemix13 virtual13 servers13 and13 allows13 SSH13 to13 access13 the13 server13

Acces13 the13 root13 user13 To13 access13 the13 root13 user13 log13 on13 as13 ibmcloud13 user13 and13 run13 the13 following13 command13

[ibmcloudelkforcase ~]$ sudo bash

[rootelkforcase ibmcloud]

The13 root13 user13 can13 be13 used13 to13 add13 programs13 to13 the13 server13 and13 create13 and13 edit13 users13 as13 required13 to13 access13 applicaons13 installed13 on13 the13 virtual13 server13

Page of 12 71 Page of 12 71

Copy13 files13 to13 the13 virtual13 server13 To13 move13 installaon13 files13 to13 the13 server13 go13 to13 the13 UNIX13 or13 Linux13 window13 you13 used13 to13 create13 the13 SSH13 key13 and13 the13 directory13 where13 the13 ltuniquekeynamegt13 files13 exist13 Move13 the13 installaon13 files13 to13 the13 same13 locaon13

Run13 the13 following13 command13 referencing13 the13 SSH13 keyname13 the13 public13 IP13 address13 Bluemix13 allocated13 for13 the13 virtual13 server13 and13 the13 password13 for13 the13 SSH13 key13

[userserver ~]$ scp ndashi ltuniquekeynamegt ltINSTALLATIONFILENAMEgt ibmcloudltpublicIPaddressgthomeibmcloud

Enter13 the13 passphrase13 for13 key13 cloudkey313

This13 copies13 the13 file13 using13 the13 ibmcloud13 user13 to13 the13 homeibmcloud13 directory13 on13 the13 virtual13 server13

Install13 and13 access13 the13 virtual13 server13 via13 VNC13 server13 Using13 the13 ssh13 command13 to13 access13 the13 virtual13 server13 is13 not13 the13 most13 stable13 method13 and13 does13 not13 offer13 X-shy‐Windows13 support13 Therefore13 we13 suggest13 that13 you13 use13 a13 VNC13 client13 to13 access13 the13 virtual13 server13

Installing13 a13 VNC13 client13 should13 be13 done13 as13 the13 root13 user13 Use13 the13 yum13 command13 as13 follows13

[ibmcloudelkforcase ~]$ yum groupinstall GNOME Desktop

[ibmcloudelkforcase ~]$ yum install tigervnc-server

These13 commands13 first13 install13 a13 desktop13 style13 OS13 user13 interface13 and13 then13 a13 VNC13 client13 to13 remotely13 access13 the13 desktop13 style13 OS13

Then13 as13 either13 the13 root13 user13 or13 another13 user13 (example13 below13 uses13 elk)13 run13 the13 following13 command13 to13 create13 the13 VNC13 client13 connecon13

[elkelkforcase ~]$ vncserver

New elkforcase2 (elk) desktop is elkforcase2

Starting applications specified in homeelkvncxstartup

Log file is homeelkvncelkforcase2log

[elkelkforcase ~]$

Install13 a13 VNC13 client13 on13 the13 client13 machine13 used13 to13 access13 the13 Internet13 and13 (in13 the13 example13 above)13 use13 the13 command13 ltpublicIPadddressgt213 and13 the13 password13 for13 ELK13 to13 access13 the13 virtual13 server13

Using13 a13 VNC13 client13 means13 that13 you13 can13 leave13 long13 duraon13 installaons13 running13 without13 fear13 of13 the13 terminal13 losing13 connecon13 and13 the13 installaons13 failing13

Install13 the13 ELK13 stack13 Download13 the13 source13 soaware13 code13 for13 the13 ELK13 stack13 from13 the13 ELK13 website13 13

At13 the13 me13 of13 wring13 the13 downloaded13 soaware13 files13 were13

Page of 13 71 Page of 13 71

13

Transfer13 the13 files13 to13 the13 virtual13 server13 (using13 the13 scp13 method13 detailed13 above)13

Install13 Elas0csearch13 As13 a13 root13 user13 create13 a13 non-shy‐root13 user13 with13 the13 adduser13 command13

Create13 a13 directory13 to13 install13 Elascsearch13 into13 and13 untar13 the13 contents13 of13 the13 Elascsearch13 install13 file13 into13 that13 directory13

[elkelkforcase]$ cd home

[elkelkforcase]$ mkdir elasticsearch

[elkelkforcase]$ cp homeibmcloudelasticsearch-233targz homeelasticsearch

[elkelkforcase]$ cd elasticsearch

[elkelkforcase]$ gunzip elasticsearch-233targz

[elkelkforcase]$ untar elasticsearch-233tar

Elascsearch13 is13 now13 installed13

Start13 Elas0csearch13 To13 start13 Elascsearch13 go13 to13 the13 bin13 directory13 as13 the13 non-shy‐root13 user13 when13 you13 installed13 it13 and13 run13 the13 elascsearch13 executable13

[elkelkforcase]$ cd homeelasticsearchelasticsearch-233bin

[elkelkforcase]$ elasticsearch amp

[2] 22211

[elkelkforcase]$ [2016-07-05 133857730][INFO ][node] [Arsenal] version[233] pid[22211] build[218bdf12016-05-17T154004Z]

[2016-07-05 133857740][INFO ][node] [Arsenal] initializing

[2016-07-05 133858984][INFO ][plugins] [Arsenal] modules [reindex lang-expression lang-groovy] plugins [] sites []

[2016-07-05 133859146][INFO ][env] [Arsenal] using [1] data paths mounts [[ (rootfs)]] net usable_space [155gb] net total_space [199gb] spins [unknown] types [rootfs]

[2016-07-05 133859146][INFO ][env] [Arsenal] heap size [10156mb] compressed ordinary object pointers [true]

Page of 14 71 Page of 14 71

[2016-07-05 133859147][WARN ][env] [Arsenal] max file descriptors [4096] for elasticsearch process likely too low consider increasing to at least [65536]

[2016-07-05 133903526][INFO ][node] [Arsenal] initialized

[2016-07-05 133903526][INFO ][node] [Arsenal] starting

To13 test13 that13 Elascsearch13 is13 running13 correctly13 open13 the13 browser13 in13 the13 VNC13 client13 and13 run13 localhost920013

13

Install13 Kibana13 ELK13 recommends13 installing13 and13 running13 Kibana13 as13 root13 13

1 As13 root13 user13 create13 an13 installaon13 directory13 and13 untar13 the13 installaon13 file13

[rootelkforcase]$ cd home

[rootelkforcase]$ mkdir kibana

[rootelkforcase]$ cp homeibmcloudkibana-451-linux-x64targz homekibana

[rootelkforcase]$ cd kibana

[rootelkforcase]$ gunzip kibana-451-linux-x64targz

[rootelkforcase]$ untar kibana-451-linux-x64tar

2 Modify13 the13 manifestyml13 configuraon13 file13 Go13 to13 the13 kibana-shy‐451-shy‐linux-shy‐x6413 directory13

[rootelkforcase]$ cd homekibanakibana-451-linux-x64

3 Edit13 the13 manifestyml13 file13 so13 the13 contents13 are13

applicaons13

-shy‐ name13 kibana-shy‐in-shy‐bluemix13

memory13 128M13

host13 kibana-shy‐in-shy‐bluemix13

buildpack13 hcpsgithubcomcloudfoundry-shy‐communitynginx-shy‐buildpackgit13

Page of 15 71 Page of 15 71

4 Edit13 the13 configkibanayml13 file13 so13 the13 contents13 are13

Kibana is served by a back end server This controls which port to use

port 5601

The host to bind the server to

host 0000

If you are running kibana behind a proxy and want to mount it at a path

specify that path here The basePath cant end in a slash

serverbasePath

The maximum payload size in bytes on incoming server requests

servermaxPayloadBytes 1048576

The Elasticsearch instance to use for all your queries

elasticsearchurl httplocalhost9200

preserve_elasticsearch_host true will send the hostname specified in `elasticsearch` If you set it to false

then the host you use to connect to this Kibana instance will be sent

elasticsearchpreserveHost true

Kibana uses an index in Elasticsearch to store saved searches visualizations

and dashboards It will create a new index if it doesnt already exist

kibanaindex kibana

The default application to load

kibanadefaultAppId discover

If your Elasticsearch is protected with basic auth these are the user credentials

used by the Kibana server to perform maintenance on the kibana_index at startup Your

Kibana users will still need to authenticate with Elasticsearch (which is proxied through

the Kibana server)

Page of 16 71 Page of 16 71

elasticsearchusername user

elasticsearchpassword pass

SSL for outgoing requests from the Kibana Server to the browser (PEM formatted)

serversslcert pathtoyourservercrt

serversslkey pathtoyourserverkey

Optional setting to validate that your Elasticsearch backend uses the same key files (PEM formatted)

elasticsearchsslcert pathtoyourclientcrt

elasticsearchsslkey pathtoyourclientkey

If you need to provide a CA certificate for your Elasticsearch instance put

the path of the pem file here

elasticsearchsslca pathtoyourCApem

Set to false to have a complete disregard for the validity of the SSL

certificate

elasticsearchsslverify true

Time in milliseconds to wait for elasticsearch to respond to pings defaults to

request_timeout setting

elasticsearchpingTimeout 1500

Time in milliseconds to wait for responses from the back end or elasticsearch

This must be gt 0

elasticsearchrequestTimeout 30000

Time in milliseconds for Elasticsearch to wait for responses from shards

Set to 0 to disable

elasticsearchshardTimeout 0

Time in milliseconds to wait for Elasticsearch at Kibana startup before retrying

elasticsearchstartupTimeout 5000

elasticsearchstartupTimeout 30000

Page of 17 71 Page of 17 71

Set the path to where you would like the process id file to be created

pidfile varrunkibanapid

If you would like to send the log output to a file you can set the path below

loggingdest stdout

Set this to true to suppress all logging output

loggingsilent false

Set this to true to suppress all logging output except for error messages

loggingquiet false

Set this to true to log all events including system usage information and all requests

loggingverbose false

Kibana13 is13 now13 installed13

Start13 Kibana13 To13 start13 Kibana13 go13 to13 the13 bin13 directory13 and13 run13 the13 kibana13 executable13

[rootelkforcase]$

[rootelkforcase]$ cd homekibanakibana-451-linux-x64bin

[rootelkforcase]$ kibana amp

[1] 22283

[rootelkforcase] log [095900542] [info][status][pluginkibana] Status changed from uninitialized to green - Ready

log [095900600] [info][status][pluginelasticsearch] Status changed from uninitialized to yellow - Waiting for Elasticsearch

log [095900689] [info][status][pluginkbn_vislib_vis_types] Status changed from uninitialized to green - Ready

log [095900717] [info][status][pluginmarkdown_vis] Status changed from uninitialized to green - Ready

log [095900721] [info][status][pluginmetric_vis] Status changed from uninitialized to green - Ready

log [095900814] [info][status][pluginspyModes] Status changed from uninitialized to green - Ready

log [095900934] [info][status][pluginstatusPage] Status changed from uninitialized to green - Ready

Page of 18 71 Page of 18 71

log [095900941] [info][status][plugintable_vis] Status changed from uninitialized to green - Ready

log [095901067] [info][listening] Server running at http00005601

log [095906083] [info][status][pluginelasticsearch] Status changed from yellow to yellow - No existing Kibana index found

log [095908942] [info][status][pluginelasticsearch] Status changed from yellow to green - Kibana index ready

Kibana13 is13 now13 started13

Broadcast13 Bluemix13 app13 log13 records13 Bluemix13 apps13 (which13 are13 shown13 in13 the13 US13 South13 Bluemix13 region13 of13 the13 System13 Architecture13 diagram)13 will13 produce13 logs13 in13 a13 syslog13 rfc542413 format13 using13 a13 Bluemix13 tool13 called13 Loggregator13

For13 Logstash13 to13 receive13 these13 logs13 each13 Bluemix13 app13 needs13 to13 broadcast13 these13 logs13 To13 do13 that13 run13 the13 following13 commands13 on13 each13 Bluemix13 app13 using13 the13 public13 IP13 address13 of13 the13 ELK13 virtual13 server13 the13 ELK13 tag13 to13 trace13 the13 process13 and13 insert13 the13 name13 of13 each13 app13 in13 turn13

cf cups elk -l syslogltpublicIPaddress5000 cf bind-service ltmyappgt elk

cf restage ltmyappgt

Install13 Logstash13 To13 install13 Logstash13 as13 root13 user13 create13 a13 directory13 for13 the13 installaon13 and13 untar13 the13 installaon13 file13 there13

13 13 [rootelkforcase]$ cd home

[rootelkforcase]$ mkdir logstash

[rootelkforcase]$ cp homeibmcloudlogstash-233targz homelogstash

[rootelkforcase]$ cd logstash

[rootelkforcase]$ gunzip logstash-233targz

[rootelkforcase]$ untar logstash-233tar

Write13 Logstash13 configura0on13 Logstash13 requires13 a13 configuraon13 file13 to13 be13 wricen13 to13 receive13 transform13 and13 output13 the13 log13 records13 to13 Elascsearch13 This13 secon13 explains13 the13 basic13 requirements13 of13 processing13 Bluemix13 app13 log13 records13 into13 Elascsearch13 (more13 complex13 configuraons13 are13 detailed13 later13 in13 this13 document13 for13 the13 integraons13 with13 third-shy‐party13 tools)13

Create13 a13 conf13 file13 as13 root13 user13

[rootelkforcase]$ cd homelogstashlogstash-233

[rootelkforcase]$ mkdir config

Page of 19 71 Page of 19 71

[rootelkforcase]$ cd config

[rootelkforcase]$ touch loggregatorconf

Edit13 the13 loggregator13 file13 with13 the13 following13 contents13

13 Input13 secon13 defines13 which13 log13 records13 to13 receive13

13 port13 and13 type13 reflect13 method13 selected13 for13 Cloud13 Foundry13 to13 broadcast13 the13 records13

input

tcp

port =gt 5000

type =gt syslog

13 Filter13 secon13 ndash13 transforms13 the13 data13

filter

Add a field to identify the records as Cloud Foundry based

mutate

add_field =gt format =gt cf

13 Output13 secon13 ndash13 sends13 the13 data13 to13 its13 desnaon(s)13

output

13 Use13 the13 elascsearch13 plugin13 to13 send13 the13 data13 to13 the13 elascsearch13 instance13

13 elasticsearch

hosts =gt localhost

stdout codec =gt rubydebug

13 End13 of13 Config13

Save13 the13 file13 Verify13 the13 format13 of13 the13 file13 is13 acceptable13 by13 running13 the13 following13 commands13

[rootelkforcase]$ cd homelogstashlogstash-233

[rootelkforcase]$ binlogstash agent --configtest -f configloggregatorconf

Page of 20 71 Page of 20 71

Configuraon13 is13 OK13

Start13 Logstash13 Aaer13 you13 have13 wricen13 and13 verified13 the13 Logstash13 configuraon13 you13 can13 start13 Logstash13 to13 process13 records13 from13 the13 Bluemix13 apps13 (via13 Cloud13 Foundry)13 Run13 the13 following13 as13 root13 user13

[rootelkforcase]$ cd homelogstashlogstash-233

[rootelkforcase]$ binlogstash agent -f configloggregatorconf

Logstash13 is13 now13 started13

Monitor13 Logstash13 processing13 Once13 Logstash13 is13 running13 and13 log13 records13 are13 being13 produced13 by13 the13 Bluemix13 apps13 you13 can13 monitor13 progress13 in13 the13 same13 window13 that13 was13 used13 to13 start13 Logstash13 Output13 is13 similar13 to13 the13 following13

13

The13 log13 records13 being13 broadcast13 from13 the13 Bluemix13 apps13 are13 now13 sent13 to13 Elascsearch13 from13 the13 Logstash13 output13 configuraon13 Records13 are13 sent13 in13 real13 me13 If13 Logstash13 is13 not13 running13 while13 log13 records13 are13 being13 produced13 then13 these13 records13 will13 not13 be13 processed13 by13 Logstash13 and13 will13 not13 be13 stored13 in13 Elascsearch13

Page of 21 71 Page of 21 71

Use13 ELK13 The13 log13 records13 from13 the13 Bluemix13 apps13 are13 now13 being13 stored13 and13 indexed13 by13 the13 Elascsearch13 component13 This13 data13 can13 be13 accessed13 via13 the13 Kibana13 applicaon13

Kibana13 Discover13 page13 Using13 a13 web13 browser13 either13 directly13 from13 a13 client13 or13 using13 the13 VNC13 client13 access13 Kibana13 using13 the13 publicIPaddress13 of13 the13 ELK13 virtual13 server13

hcpltpublicIPaddressgt560113

No13 userpassword13 credenals13 are13 required13

Kibana13 defaults13 to13 a13 search13 of13 all13 records13 ()13 in13 the13 last13 1513 minutes13 and13 returns13 the13 results13

13

More13 complex13 searches13 are13 possible13 by13 changing13 the13 me13 filter13 and13 supplying13 search13 queries13

Change13 the13 0me13 filter13 To13 change13 the13 me13 ranges13 click13 the13 clock13 symbol13 in13 the13 top-shy‐right13 of13 the13 Kibana13 Discover13 page13

13

You13 will13 see13 the13 following13 opons13

13

Page of 22 71 Page of 22 71

A13 variety13 of13 quick13 me13 ranges13 can13 be13 selected13 or13 you13 can13 switch13 to13 relave13 or13 absolute13 me13 ranges13

Quick13 me13 ranges13 allow13 switching13 between13 last13 X13 minutes13 to13 last13 Y13 hours13 or13 Z13 days13 etc13

Relave13 me13 ranges13 allow13 more13 specific13 versions13 of13 the13 Quick13 opons13 from13 specific13 minuteshoursdays13 to13 now13

13

Absolute13 me13 ranges13 specify13 the13 start13 and13 end13 mes13 for13 the13 searches13

13

Custom13 search13 queries13 You13 can13 use13 the13 text13 window13 of13 the13 Discover13 page13 to13 search13 for13 matches13 within13 the13 log13 records13 You13 can13 use13 Apache13 Lucene13 or13 Elascsearch13 query13 DSL13 syntax13

13

Page of 23 71 Page of 23 71

Saved13 searches13 You13 can13 save13 any13 custom13 search13 syntax13 and13 choice13 of13 me13 range13 for13 future13 ease13 of13 use13 Any13 relave13 or13 absolute13 me13 frame13 is13 saved13 as13 part13 of13 the13 search13 13

To13 save13 a13 search13

1 Set13 up13 and13 run13 the13 required13 syntax13 and13 me13 frame13

13

2 Click13 the13 Save13 Search13 icon13

13

3 Type13 in13 a13 name13 for13 the13 saved13 search13

Page of 24 71 Page of 24 71

13

7 To13 delete13 or13 update13 saved13 searches13 click13 13 manage13 saved13 searches13

13

Use13 the13 edit13 icons13 to13 make13 any13 changes13

Page of 26 71 Page of 26 71

13

Build13 simple13 Kibana13 histograms13 Kibana13 allows13 you13 to13 build13 charts13 and13 dashboards13 based13 on13 the13 search13 capability13 offered13 by13 the13 Discover13 page13 Saved13 searches13 can13 be13 used13 as13 the13 basis13 for13 charts13 and13 dashboards13 built13 using13 those13 charts13 To13 build13 a13 simple13 histogram13

1 Select13 the13 Visualize13 page13

13

2 Select13 Ver0cal13 Bar13 Chart13

13

3 To13 use13 a13 saved13 search13 click13 From13 a13 saved13 search13

Page of 27 71 Page of 27 71

13

4 Select13 a13 previously13 saved13 search13

13

5 Select13 to13 edit13 the13 x-shy‐axis

6 Edit13 the13 fields13 with13 informaon13 about13 the13 aggregaon13 field13 interval13 and13 custom13 label13

Page of 28 71 Page of 28 71

13

Page of 29 71 Page of 29 71

7 Click13 the13 green13 arrow13 The13 histogram13 is13 created13 and13 run13 based13 on13 the13 saved13 search

13

8 To13 save13 the13 histogram13 click13 the13 Save13 Visualiza0on13 icon13 and13 add13 a13 tle13

13

13

9 Click13 Save13

10 To13 view13 the13 visualisaon13 click13 the13 Dashboard13 page13

Page of 30 71 Page of 30 71

13

11 Click13 the13 Add13 Visualiza0on13 icon13

13

12 Select13 the13 visiualizaon13 that13 you13 just13 created13

13

Page of 31 71 Page of 31 71

13

13 To13 save13 the13 dashboard13 click13 the13 Save13 Dashboard13 icon13

Page of 32 71 Page of 32 71

13

14 Add13 a13 tle13 and13 select13 whether13 to13 store13 the13 selected13 me13 frame13 with13 the13 dashboard13

13

15 Click13 Save13

16 You13 can13 edit13 the13 shape13 and13 number13 of13 charts13 and13 save13 the13 dashboard13

17 To13 re-shy‐run13 the13 dashboard13 select13 the13 Dashboard13 page13 and13 click13 the13 Load13 Saved13 Dashboard13 icon13

Page of 33 71 Page of 33 71

13

18 Select13 the13 saved13 dashboard13 from13 the13 list13

13

19 The13 dashboard13 runs13

13

Page of 34 71 Page of 34 71

Page of 35 71 Page of 35 71

Aler0ng13 with13 ELK13 ELK13 examines13 the13 log13 records13 in13 process13 to13 send13 alerts13 to13 third-shy‐party13 alerng13 event13 management13 and13 social13 media13 applicaons13 As13 shown13 previously13 in13 the13 ELK13 architecture13 the13 ELK13 strategy13 for13 alerng13 is13 to13 send13 alerts13 from13 the13 Logstash13 component13 directly13 Therefore13 log13 records13 and13 alerts13 are13 sent13 to13 both13 Elascsearch13 and13 third-shy‐party13 tools13 at13 the13 same13 instant13 using13 the13 output13 secon13 of13 the13 Logstash13 configuraon13 Any13 alerts13 are13 generated13 during13 the13 filter13 secon13 processing13 of13 the13 Logstash13 configuraon13

Create13 alerts13 in13 Logstash13 The13 Logstash13 filter13 secon13 is13 used13 to13 trigger13 alerts13 based13 on13 the13 log13 records13 being13 processed13 For13 example13 log13 records13 can13 be13 checked13 for13 errors13 warnings13 crashed13 or13 stopped13 messages13 and13 generate13 an13 alert13

Here13 is13 an13 example13 Logstash13 filter13 code13

13 if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

The13 specific13 tags13 are13 set13 against13 the13 log13 records13 to13 trigger13 the13 alert13 later13 in13 the13 configuraon13 in13 the13 output13 secon13

Control13 the13 number13 of13 generated13 alerts13 using13 the13 Logstash13 throcle13 plugin13 so13 that13 either13 a13 lot13 of13 instances13 in13 a13 fixed13 me13 period13 are13 required13 before13 an13 alert13 is13 sent13 or13 a13 specific13 number13 of13 instances13 triggers13 the13 alert13

Page of 36 71 Page of 36 71

Edit13 the13 plugin13 parameters13 to13 meet13 system13 requirements13

13 13 13 13 13 if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 10

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle13

after_count =gt 20

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 30

period =gt 1200

key =gt servicename_alarm_warning

Page of 37 71 Page of 37 71

add_tag =gt servicename_throttled

13 13 13 13 13 13 13 13 13

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 40

period =gt 1500

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

Enrich13 alerts13 with13 message13 detail13 Some13 of13 the13 third-shy‐party13 integraons13 benefit13 from13 having13 components13 of13 the13 raw13 log13 record13 sent13 in13 addion13 to13 the13 raw13 record13 or13 fields13 in13 the13 record13 edited13 transformed13 or13 translated13

Grok13 Logstash13 allows13 the13 log13 record13 to13 be13 broken13 up13 into13 logical13 secons13 and13 each13 secon13 stored13 in13 a13 new13 parameter13 using13 the13 grok13 parameter13 in13 the13 filter13 secon13 Here13 is13 an13 example13 grok13

13 grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

To13 build13 the13 grok13 statement13 Logstash13 offers13 an13 online13 tool13 at13 hcpsgrokdebugherokuappcom13

Translate13 If13 specific13 fields13 in13 the13 log13 record13 are13 coded13 (for13 example13 the13 Bluemix13 app13 name13 is13 oaen13 not13 sent13 in13 the13 syslog13 messages13 instead13 the13 GUID13 is13 sent13 which13 is13 not13 useful)13 then13 the13 filter13 secon13 translate13 plugin13 can13 be13 used13 to13 add13 new13 fields13 by13 holding13 the13 decoded13 versions13 of13 those13 values13

13 translate

field =gt data1

destination =gt data1

override =gt true

Page of 38 71 Page of 38 71

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

dc50f6e4-c052-436a-bf27-95ba392c4228 monverifyservicemanagement

0c677e82-5bfc-4e38-9155-e416249e3ee6 ms-catalog-postpyretic-pomiculture

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

However13 the13 translate13 plugin13 is13 not13 included13 with13 the13 Logstash13 installaon13 In13 the13 next13 secon13 I13 will13 13 describe13 how13 to13 add13 new13 plugins13 to13 the13 Logstash13 instance13

Add13 custom13 plugins13 to13 Logstash13 As13 an13 open13 source13 tool13 Logstash13 encourages13 users13 to13 develop13 content13 for13 all13 users13 to13 share13 Some13 of13 the13 plugins13 that13 were13 created13 by13 our13 community13 include13 a13 translaon13 plugin13 an13 output13 Slack13 plugin13 and13 an13 output13 syslog13 plugin13 Letrsquos13 look13 at13 how13 to13 add13 these13 plugins13 to13 your13 environment13

Filter13 translate13 plugin13 To13 add13 the13 translate13 plugin13

1 Download13 the13 plugin13

2 Copy13 the13 plugin13 file13 logstash-shy‐filter-shy‐translate-shy‐214gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-filter-translate 214

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Output13 Slack13 plugin13 To13 add13 the13 Slack13 plugin13

Page of 39 71 Page of 39 71

1 Download13 the13 plugin13 13

2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐slack-shy‐014gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-output-slack 014

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Output13 syslog13 plugin13 To13 add13 the13 syslog13 plugin13

1 Download13 the13 plugin13 13

2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐syslog-shy‐214gem13 to13 the13 ELK13 Virtual13 Server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-output-syslog 214

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Integrate13 Logstash13 with13 Slack13 To13 send13 generated13 alerts13 to13 Slack13 complete13 these13 steps13

1 Create13 a13 Slack13 integraon13 13

2 Use13 the13 Slack13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 Slack13 integraon13

113 Login13 to13 Slack13 and13 go13 to13 the13 Apps13 page13 13

Page of 40 71 Page of 40 71

13

313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13

13

A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13

13

Page of 42 71 Page of 42 71

The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13

13

The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13

13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

slack

url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5

channel =gt cloud-operations-mon

username =gt nick_cawood

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13

Page of 43 71 Page of 43 71

Page of 44 71 Page of 44 71

In13 Slack13 the13 alerts13 look13 like13 this13

13

Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13

1 Create13 a13 PagerDuty13 integraon13 13

2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13

1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13

13

2 Click13 Add13 New13 Service13

13

313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13

Page of 45 71 Page of 45 71

13

413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13

513 Click13 Add13 Service13

13

The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13

Page of 46 71 Page of 46 71

13

The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt be7eb1ea9c224018923e818d3e891d0b

incident_key =gt logstashhosttype

Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13

Page of 47 71 Page of 47 71

13

Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13

1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13

2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13

3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13

4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13

13 13 http

url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo

http_method =gt post

format =gt json

For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13

13 13 syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

Page of 48 71 Page of 48 71

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13

13

The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13

if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2

This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13

Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13

Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13

Logstash13 severity Shown13 in13 Omnibus13 as

alert Indeterminate

crical Warning

debug 7

emergency Clear

error Minor

informaonal 6

noce Crical

warning Major

Page of 49 71 Page of 49 71

13

Page of 50 71 Page of 50 71

Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13

input

tcp

port =gt 5000

type =gt syslog

filter

mutate

add_field =gt format =gt cf

grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

translate

field =gt data1

destination =gt data1

override =gt true

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

=== Check Alarm Criteria ===

Standard Key Words

if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

=== Alarm Exclusions ===

if [message] =~ (i)sometexthere

mutate remove_tag =gt servicename_alarm

Throttle alarms so we do not hit PD rate limits

Page of 52 71 Page of 52 71

if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 5

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle

after_count =gt 5

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 5

period =gt 1200

key =gt servicename_alarm_warning

add_tag =gt servicename_throttled

Page of 53 71 Page of 53 71

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

output

elasticsearch

hosts =gt localhost

stdout codec =gt rubydebug

if servicename_alarm_err in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

Page of 54 71 Page of 54 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 55 71 Page of 55 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

Page of 56 71 Page of 56 71

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert err message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_error in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

Page of 57 71 Page of 57 71

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 58 71 Page of 58 71

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 59 71 Page of 59 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert error message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

Page of 60 71 Page of 60 71

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

Page of 61 71 Page of 61 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 62 71 Page of 62 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt x

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert fatal message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

Page of 63 71 Page of 63 71

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

Page of 64 71 Page of 64 71

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 65 71 Page of 65 71

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt heavy_exclamation_mark

Page of 66 71 Page of 66 71

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert WARNING message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 67 71 Page of 67 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

Page of 68 71 Page of 68 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 69 71 Page of 69 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

Page of 70 71 Page of 70 71

Page of 71 71 Page of 71 71

Page 6: How to integrate the Elasticsearch Logstash Kibana (ELK ... · How to integrate the Elasticsearch Logstash Kibana (ELK) log analytics stack into IBM Bluemix Nick Cawood IBM Cloud

Service13 Management13 architecture13 The13 IBM13 Cloud13 Architecture13 Center13 created13 a13 standard13 reference13 architecture13 for13 how13 to13 set13 up13 your13 cloud13 environemnt13 to13 handle13 incident13 management13 13

View13 the13 standard13 Service13 Management13 architecure13

13 13

Log13 Monitoring13 System13 architecture13 The13 following13 diagram13 shows13 a13 standard13 architecture13 for13 the13 log-shy‐monitoring13 system13 described13 in13 this13 tutorial13 13

13

Bluemix13 environment13 system13 architecture13 The13 ELK13 stack13 can13 be13 installed13 into13 a13 variety13 of13 environments13 When13 using13 Bluemix13 the13 following13 figure13 shows13 what13 the13 environment13 might13 look13 like13

Page of 6 71 Page of 6 71

13

The13 Bluemix13 cloud13 plaRorm13 offers13 a13 Public13 network13 and13 access13 to13 US13 South13 and13 UK13 environments13 From13 all13 these13 environments13 third-shy‐party13 cloud-shy‐based13 soluons13 are13 also13 accessible13

The13 ELK13 stack13 is13 installed13 in13 the13 Bluemix13 United13 Kingdom13 environment13 because13 this13 is13 the13 environment13 where13 Bluemix13 has13 virtual13 servers13 available13 for13 provisioning13 Other13 applicaons13 in13 the13 CASE13 project13 were13 also13 installed13 on13 virtual13 servers13 in13 the13 Bluemix13 United13 Kingdom13 region13

Bluemix13 apps13 were13 used13 to13 create13 a13 system13 to13 be13 monitored13 by13 ELK13 These13 apps13 were13 hosted13 in13 the13 Bluemix13 US13 South13 environment13 using13 Bluemix13 microservices13 which13 produce13 logs13 during13 normal13 operaon13 and13 during13 crashes13 and13 outages13

Through13 the13 Bluemix13 Cloud13 Foundry13 process13 the13 log13 records13 can13 be13 broadcast13 from13 the13 Bluemix13 apps13 so13 Logstash13 can13 receive13 them13 and13 send13 them13 to13 Elascsearch13 within13 the13 ELK13 stack13 or13 other13 virtual13 servers13 (hosng13 other13 IBM13 or13 third-shy‐party13 products)13 in13 the13 Bluemix13 United13 Kingdom13 environment13 or13 to13 third-shy‐party13 applicaons13 in13 the13 public13 cloud13 (Pager13 Duty13 Slack13 etc)13

Crea0ng13 a13 virtual13 server13 in13 Bluemix13 With13 Bluemix13 you13 can13 provision13 a13 virtual13 server13 in13 the13 cloud13 to13 install13 standalone13 or13 integrated13 system13 components13 Not13 all13 Bluemix13 regions13 offer13 this13 service13 all13 the13 me13 due13 to13 beta-shy‐project13 constraints13 In13 this13 arclersquos13 examples13 virtual13 servers13 were13 allowed13 to13 be13 provisioned13 in13 the13 United13 Kingdom13 region13 For13 more13 informaon13 read13 how13 to13 provision13 a13 Bluemix13 virtual13 server13 including13 some13 simple13 setup13 steps13 for13 ease13 of13 use13

Page of 7 71 Page of 7 71

Create13 an13 SSH13 key13 To13 access13 your13 virtual13 server13 once13 it13 is13 provisioned13 allocate13 an13 SSH13 key13 to13 the13 virtual13 server13

To13 generate13 a13 publicprivate13 RSA13 key13 pair13 follow13 these13 steps13

1 In13 a13 UNIX13 or13 Linuxreg13 window13 run13 the13 following13 command13 specifying13 a13 ltuniquekeynamegt13 of13 your13 your13 choice13 and13 a13 password13 or13 passphrase13 (memorize13 the13 password)$13 ssh-shy‐keygen13 -shy‐t13 rsa13 -shy‐f13 ltuniquekeynamegt13

2 Enter13 the13 passphrase13 (empty13 for13 no13 passphrase)13

3 Enter13 the13 same13 passphrase13 again13

Your13 idenficaon13 has13 been13 saved13 in13 ltuniquekeynamegt13

Your13 public13 key13 has13 been13 saved13 in13 ltuniquekeynamegtpub13

The13 key13 fingerprint13 is13

d6268464506525930ce4632c693c203c13 ltusergtltservergt13

The13 keys13 randomart13 image13 is13

+--[ RSA 2048]----+

|o ++=o |

| E o ooo |

| |

| + o |

| S o |

| o |

| |

| |

| |

+-----------------+

Two13 keys13 are13 created13 public13 and13 private13 The13 public13 key13 contents13 are13 used13 to13 create13 a13 virtual13 server13 with13 a13 public13 IP13 address13 The13 private13 key13 is13 used13 to13 access13 the13 virtual13 server13 from13 a13 client13 machine13

Page of 8 71 Page of 8 71

Provisioning13 a13 virtual13 server13 1 A13 Bluemix13 account13 is13 required13 for13 this13 process13 Sign13 up13 here13 and13 log13 in13 Make13 sure13 your13

environment13 is13 correct13 For13 example13 if13 you13 are13 in13 the13 United13 Kingdom13 select13 the13 United13 Kingdom13 region13 in13 the13 Bluemix13 console13

13

13

2 Add13 a13 virtual13 server13 to13 your13 Bluemix13 environment13 Click13 Run13 Virtual13 Servers13

13

3 Select13 an13 OS13 image13 from13 the13 EXISTING13 list13 For13 ELK13 select13 Centos13 OS13 (use13 latest13 available13 version)13

Page of 9 71 Page of 9 71

4 Name13 the13 virtual13 server13

5 Make13 sure13 Assign13 public13 IP13 address13 is13 checked13

13

6 Click13 Add13 Key13 to13 add13 the13 SSH13 key13 you13 created13 previously13 Copy13 the13 contents13 of13 the13 public13 key13 into13 the13 Public13 Key13 to13 import13 field13

Page of 10 71 Page of 10 71

13

7 Click13 OK13 Then13 click13 CREATE13

The13 virtual13 server13 is13 provisioned13 and13 added13 to13 the13 Bluemix13 dashboard13 under13 VIRTUAL13 SERVERS13

Page of 11 71 Page of 11 71

13

Set13 up13 the13 virtual13 server13

Log13 on13 to13 the13 virtual13 server13 Now13 you13 can13 log13 on13 to13 the13 virtual13 server13 and13 add13 new13 users13 and13 programs13 to13 make13 it13 easier13 to13 access13 the13 virtual13 server13

To13 access13 the13 server13 go13 to13 the13 UNIX13 or13 Linux13 window13 that13 you13 used13 to13 create13 the13 SSH13 key13 In13 the13 directory13 where13 the13 ltuniquekeynamegt13 files13 exist13 run13 the13 following13 command13 referencing13 the13 SSH13 keyname13 the13 public13 IP13 address13 Bluemix13 allocated13 for13 the13 virtual13 server13 and13 the13 password13 for13 the13 SSH13 key13

[userserver ~]$ ssh -i ltuniquekeynamegt ibmcloudltpublicIPaddressgt

Enter13 the13 passphrase13 for13 the13 key13 ltuniquekeynamegt13

Last login Tue Jul 5 125546 2016 from 108-210-96-231lightspeedrlghncsbcglobalnet

[ibmcloudelkforcase ~]$

ibmcloud13 is13 the13 default13 user13 for13 the13 Bluemix13 virtual13 servers13 and13 allows13 SSH13 to13 access13 the13 server13

Acces13 the13 root13 user13 To13 access13 the13 root13 user13 log13 on13 as13 ibmcloud13 user13 and13 run13 the13 following13 command13

[ibmcloudelkforcase ~]$ sudo bash

[rootelkforcase ibmcloud]

The13 root13 user13 can13 be13 used13 to13 add13 programs13 to13 the13 server13 and13 create13 and13 edit13 users13 as13 required13 to13 access13 applicaons13 installed13 on13 the13 virtual13 server13

Page of 12 71 Page of 12 71

Copy13 files13 to13 the13 virtual13 server13 To13 move13 installaon13 files13 to13 the13 server13 go13 to13 the13 UNIX13 or13 Linux13 window13 you13 used13 to13 create13 the13 SSH13 key13 and13 the13 directory13 where13 the13 ltuniquekeynamegt13 files13 exist13 Move13 the13 installaon13 files13 to13 the13 same13 locaon13

Run13 the13 following13 command13 referencing13 the13 SSH13 keyname13 the13 public13 IP13 address13 Bluemix13 allocated13 for13 the13 virtual13 server13 and13 the13 password13 for13 the13 SSH13 key13

[userserver ~]$ scp ndashi ltuniquekeynamegt ltINSTALLATIONFILENAMEgt ibmcloudltpublicIPaddressgthomeibmcloud

Enter13 the13 passphrase13 for13 key13 cloudkey313

This13 copies13 the13 file13 using13 the13 ibmcloud13 user13 to13 the13 homeibmcloud13 directory13 on13 the13 virtual13 server13

Install13 and13 access13 the13 virtual13 server13 via13 VNC13 server13 Using13 the13 ssh13 command13 to13 access13 the13 virtual13 server13 is13 not13 the13 most13 stable13 method13 and13 does13 not13 offer13 X-shy‐Windows13 support13 Therefore13 we13 suggest13 that13 you13 use13 a13 VNC13 client13 to13 access13 the13 virtual13 server13

Installing13 a13 VNC13 client13 should13 be13 done13 as13 the13 root13 user13 Use13 the13 yum13 command13 as13 follows13

[ibmcloudelkforcase ~]$ yum groupinstall GNOME Desktop

[ibmcloudelkforcase ~]$ yum install tigervnc-server

These13 commands13 first13 install13 a13 desktop13 style13 OS13 user13 interface13 and13 then13 a13 VNC13 client13 to13 remotely13 access13 the13 desktop13 style13 OS13

Then13 as13 either13 the13 root13 user13 or13 another13 user13 (example13 below13 uses13 elk)13 run13 the13 following13 command13 to13 create13 the13 VNC13 client13 connecon13

[elkelkforcase ~]$ vncserver

New elkforcase2 (elk) desktop is elkforcase2

Starting applications specified in homeelkvncxstartup

Log file is homeelkvncelkforcase2log

[elkelkforcase ~]$

Install13 a13 VNC13 client13 on13 the13 client13 machine13 used13 to13 access13 the13 Internet13 and13 (in13 the13 example13 above)13 use13 the13 command13 ltpublicIPadddressgt213 and13 the13 password13 for13 ELK13 to13 access13 the13 virtual13 server13

Using13 a13 VNC13 client13 means13 that13 you13 can13 leave13 long13 duraon13 installaons13 running13 without13 fear13 of13 the13 terminal13 losing13 connecon13 and13 the13 installaons13 failing13

Install13 the13 ELK13 stack13 Download13 the13 source13 soaware13 code13 for13 the13 ELK13 stack13 from13 the13 ELK13 website13 13

At13 the13 me13 of13 wring13 the13 downloaded13 soaware13 files13 were13

Page of 13 71 Page of 13 71

13

Transfer13 the13 files13 to13 the13 virtual13 server13 (using13 the13 scp13 method13 detailed13 above)13

Install13 Elas0csearch13 As13 a13 root13 user13 create13 a13 non-shy‐root13 user13 with13 the13 adduser13 command13

Create13 a13 directory13 to13 install13 Elascsearch13 into13 and13 untar13 the13 contents13 of13 the13 Elascsearch13 install13 file13 into13 that13 directory13

[elkelkforcase]$ cd home

[elkelkforcase]$ mkdir elasticsearch

[elkelkforcase]$ cp homeibmcloudelasticsearch-233targz homeelasticsearch

[elkelkforcase]$ cd elasticsearch

[elkelkforcase]$ gunzip elasticsearch-233targz

[elkelkforcase]$ untar elasticsearch-233tar

Elascsearch13 is13 now13 installed13

Start13 Elas0csearch13 To13 start13 Elascsearch13 go13 to13 the13 bin13 directory13 as13 the13 non-shy‐root13 user13 when13 you13 installed13 it13 and13 run13 the13 elascsearch13 executable13

[elkelkforcase]$ cd homeelasticsearchelasticsearch-233bin

[elkelkforcase]$ elasticsearch amp

[2] 22211

[elkelkforcase]$ [2016-07-05 133857730][INFO ][node] [Arsenal] version[233] pid[22211] build[218bdf12016-05-17T154004Z]

[2016-07-05 133857740][INFO ][node] [Arsenal] initializing

[2016-07-05 133858984][INFO ][plugins] [Arsenal] modules [reindex lang-expression lang-groovy] plugins [] sites []

[2016-07-05 133859146][INFO ][env] [Arsenal] using [1] data paths mounts [[ (rootfs)]] net usable_space [155gb] net total_space [199gb] spins [unknown] types [rootfs]

[2016-07-05 133859146][INFO ][env] [Arsenal] heap size [10156mb] compressed ordinary object pointers [true]

Page of 14 71 Page of 14 71

[2016-07-05 133859147][WARN ][env] [Arsenal] max file descriptors [4096] for elasticsearch process likely too low consider increasing to at least [65536]

[2016-07-05 133903526][INFO ][node] [Arsenal] initialized

[2016-07-05 133903526][INFO ][node] [Arsenal] starting

To13 test13 that13 Elascsearch13 is13 running13 correctly13 open13 the13 browser13 in13 the13 VNC13 client13 and13 run13 localhost920013

13

Install13 Kibana13 ELK13 recommends13 installing13 and13 running13 Kibana13 as13 root13 13

1 As13 root13 user13 create13 an13 installaon13 directory13 and13 untar13 the13 installaon13 file13

[rootelkforcase]$ cd home

[rootelkforcase]$ mkdir kibana

[rootelkforcase]$ cp homeibmcloudkibana-451-linux-x64targz homekibana

[rootelkforcase]$ cd kibana

[rootelkforcase]$ gunzip kibana-451-linux-x64targz

[rootelkforcase]$ untar kibana-451-linux-x64tar

2 Modify13 the13 manifestyml13 configuraon13 file13 Go13 to13 the13 kibana-shy‐451-shy‐linux-shy‐x6413 directory13

[rootelkforcase]$ cd homekibanakibana-451-linux-x64

3 Edit13 the13 manifestyml13 file13 so13 the13 contents13 are13

applicaons13

-shy‐ name13 kibana-shy‐in-shy‐bluemix13

memory13 128M13

host13 kibana-shy‐in-shy‐bluemix13

buildpack13 hcpsgithubcomcloudfoundry-shy‐communitynginx-shy‐buildpackgit13

Page of 15 71 Page of 15 71

4 Edit13 the13 configkibanayml13 file13 so13 the13 contents13 are13

Kibana is served by a back end server This controls which port to use

port 5601

The host to bind the server to

host 0000

If you are running kibana behind a proxy and want to mount it at a path

specify that path here The basePath cant end in a slash

serverbasePath

The maximum payload size in bytes on incoming server requests

servermaxPayloadBytes 1048576

The Elasticsearch instance to use for all your queries

elasticsearchurl httplocalhost9200

preserve_elasticsearch_host true will send the hostname specified in `elasticsearch` If you set it to false

then the host you use to connect to this Kibana instance will be sent

elasticsearchpreserveHost true

Kibana uses an index in Elasticsearch to store saved searches visualizations

and dashboards It will create a new index if it doesnt already exist

kibanaindex kibana

The default application to load

kibanadefaultAppId discover

If your Elasticsearch is protected with basic auth these are the user credentials

used by the Kibana server to perform maintenance on the kibana_index at startup Your

Kibana users will still need to authenticate with Elasticsearch (which is proxied through

the Kibana server)

Page of 16 71 Page of 16 71

elasticsearchusername user

elasticsearchpassword pass

SSL for outgoing requests from the Kibana Server to the browser (PEM formatted)

serversslcert pathtoyourservercrt

serversslkey pathtoyourserverkey

Optional setting to validate that your Elasticsearch backend uses the same key files (PEM formatted)

elasticsearchsslcert pathtoyourclientcrt

elasticsearchsslkey pathtoyourclientkey

If you need to provide a CA certificate for your Elasticsearch instance put

the path of the pem file here

elasticsearchsslca pathtoyourCApem

Set to false to have a complete disregard for the validity of the SSL

certificate

elasticsearchsslverify true

Time in milliseconds to wait for elasticsearch to respond to pings defaults to

request_timeout setting

elasticsearchpingTimeout 1500

Time in milliseconds to wait for responses from the back end or elasticsearch

This must be gt 0

elasticsearchrequestTimeout 30000

Time in milliseconds for Elasticsearch to wait for responses from shards

Set to 0 to disable

elasticsearchshardTimeout 0

Time in milliseconds to wait for Elasticsearch at Kibana startup before retrying

elasticsearchstartupTimeout 5000

elasticsearchstartupTimeout 30000

Page of 17 71 Page of 17 71

Set the path to where you would like the process id file to be created

pidfile varrunkibanapid

If you would like to send the log output to a file you can set the path below

loggingdest stdout

Set this to true to suppress all logging output

loggingsilent false

Set this to true to suppress all logging output except for error messages

loggingquiet false

Set this to true to log all events including system usage information and all requests

loggingverbose false

Kibana13 is13 now13 installed13

Start13 Kibana13 To13 start13 Kibana13 go13 to13 the13 bin13 directory13 and13 run13 the13 kibana13 executable13

[rootelkforcase]$

[rootelkforcase]$ cd homekibanakibana-451-linux-x64bin

[rootelkforcase]$ kibana amp

[1] 22283

[rootelkforcase] log [095900542] [info][status][pluginkibana] Status changed from uninitialized to green - Ready

log [095900600] [info][status][pluginelasticsearch] Status changed from uninitialized to yellow - Waiting for Elasticsearch

log [095900689] [info][status][pluginkbn_vislib_vis_types] Status changed from uninitialized to green - Ready

log [095900717] [info][status][pluginmarkdown_vis] Status changed from uninitialized to green - Ready

log [095900721] [info][status][pluginmetric_vis] Status changed from uninitialized to green - Ready

log [095900814] [info][status][pluginspyModes] Status changed from uninitialized to green - Ready

log [095900934] [info][status][pluginstatusPage] Status changed from uninitialized to green - Ready

Page of 18 71 Page of 18 71

log [095900941] [info][status][plugintable_vis] Status changed from uninitialized to green - Ready

log [095901067] [info][listening] Server running at http00005601

log [095906083] [info][status][pluginelasticsearch] Status changed from yellow to yellow - No existing Kibana index found

log [095908942] [info][status][pluginelasticsearch] Status changed from yellow to green - Kibana index ready

Kibana13 is13 now13 started13

Broadcast13 Bluemix13 app13 log13 records13 Bluemix13 apps13 (which13 are13 shown13 in13 the13 US13 South13 Bluemix13 region13 of13 the13 System13 Architecture13 diagram)13 will13 produce13 logs13 in13 a13 syslog13 rfc542413 format13 using13 a13 Bluemix13 tool13 called13 Loggregator13

For13 Logstash13 to13 receive13 these13 logs13 each13 Bluemix13 app13 needs13 to13 broadcast13 these13 logs13 To13 do13 that13 run13 the13 following13 commands13 on13 each13 Bluemix13 app13 using13 the13 public13 IP13 address13 of13 the13 ELK13 virtual13 server13 the13 ELK13 tag13 to13 trace13 the13 process13 and13 insert13 the13 name13 of13 each13 app13 in13 turn13

cf cups elk -l syslogltpublicIPaddress5000 cf bind-service ltmyappgt elk

cf restage ltmyappgt

Install13 Logstash13 To13 install13 Logstash13 as13 root13 user13 create13 a13 directory13 for13 the13 installaon13 and13 untar13 the13 installaon13 file13 there13

13 13 [rootelkforcase]$ cd home

[rootelkforcase]$ mkdir logstash

[rootelkforcase]$ cp homeibmcloudlogstash-233targz homelogstash

[rootelkforcase]$ cd logstash

[rootelkforcase]$ gunzip logstash-233targz

[rootelkforcase]$ untar logstash-233tar

Write13 Logstash13 configura0on13 Logstash13 requires13 a13 configuraon13 file13 to13 be13 wricen13 to13 receive13 transform13 and13 output13 the13 log13 records13 to13 Elascsearch13 This13 secon13 explains13 the13 basic13 requirements13 of13 processing13 Bluemix13 app13 log13 records13 into13 Elascsearch13 (more13 complex13 configuraons13 are13 detailed13 later13 in13 this13 document13 for13 the13 integraons13 with13 third-shy‐party13 tools)13

Create13 a13 conf13 file13 as13 root13 user13

[rootelkforcase]$ cd homelogstashlogstash-233

[rootelkforcase]$ mkdir config

Page of 19 71 Page of 19 71

[rootelkforcase]$ cd config

[rootelkforcase]$ touch loggregatorconf

Edit13 the13 loggregator13 file13 with13 the13 following13 contents13

13 Input13 secon13 defines13 which13 log13 records13 to13 receive13

13 port13 and13 type13 reflect13 method13 selected13 for13 Cloud13 Foundry13 to13 broadcast13 the13 records13

input

tcp

port =gt 5000

type =gt syslog

13 Filter13 secon13 ndash13 transforms13 the13 data13

filter

Add a field to identify the records as Cloud Foundry based

mutate

add_field =gt format =gt cf

13 Output13 secon13 ndash13 sends13 the13 data13 to13 its13 desnaon(s)13

output

13 Use13 the13 elascsearch13 plugin13 to13 send13 the13 data13 to13 the13 elascsearch13 instance13

13 elasticsearch

hosts =gt localhost

stdout codec =gt rubydebug

13 End13 of13 Config13

Save13 the13 file13 Verify13 the13 format13 of13 the13 file13 is13 acceptable13 by13 running13 the13 following13 commands13

[rootelkforcase]$ cd homelogstashlogstash-233

[rootelkforcase]$ binlogstash agent --configtest -f configloggregatorconf

Page of 20 71 Page of 20 71

Configuraon13 is13 OK13

Start13 Logstash13 Aaer13 you13 have13 wricen13 and13 verified13 the13 Logstash13 configuraon13 you13 can13 start13 Logstash13 to13 process13 records13 from13 the13 Bluemix13 apps13 (via13 Cloud13 Foundry)13 Run13 the13 following13 as13 root13 user13

[rootelkforcase]$ cd homelogstashlogstash-233

[rootelkforcase]$ binlogstash agent -f configloggregatorconf

Logstash13 is13 now13 started13

Monitor13 Logstash13 processing13 Once13 Logstash13 is13 running13 and13 log13 records13 are13 being13 produced13 by13 the13 Bluemix13 apps13 you13 can13 monitor13 progress13 in13 the13 same13 window13 that13 was13 used13 to13 start13 Logstash13 Output13 is13 similar13 to13 the13 following13

13

The13 log13 records13 being13 broadcast13 from13 the13 Bluemix13 apps13 are13 now13 sent13 to13 Elascsearch13 from13 the13 Logstash13 output13 configuraon13 Records13 are13 sent13 in13 real13 me13 If13 Logstash13 is13 not13 running13 while13 log13 records13 are13 being13 produced13 then13 these13 records13 will13 not13 be13 processed13 by13 Logstash13 and13 will13 not13 be13 stored13 in13 Elascsearch13

Page of 21 71 Page of 21 71

Use13 ELK13 The13 log13 records13 from13 the13 Bluemix13 apps13 are13 now13 being13 stored13 and13 indexed13 by13 the13 Elascsearch13 component13 This13 data13 can13 be13 accessed13 via13 the13 Kibana13 applicaon13

Kibana13 Discover13 page13 Using13 a13 web13 browser13 either13 directly13 from13 a13 client13 or13 using13 the13 VNC13 client13 access13 Kibana13 using13 the13 publicIPaddress13 of13 the13 ELK13 virtual13 server13

hcpltpublicIPaddressgt560113

No13 userpassword13 credenals13 are13 required13

Kibana13 defaults13 to13 a13 search13 of13 all13 records13 ()13 in13 the13 last13 1513 minutes13 and13 returns13 the13 results13

13

More13 complex13 searches13 are13 possible13 by13 changing13 the13 me13 filter13 and13 supplying13 search13 queries13

Change13 the13 0me13 filter13 To13 change13 the13 me13 ranges13 click13 the13 clock13 symbol13 in13 the13 top-shy‐right13 of13 the13 Kibana13 Discover13 page13

13

You13 will13 see13 the13 following13 opons13

13

Page of 22 71 Page of 22 71

A13 variety13 of13 quick13 me13 ranges13 can13 be13 selected13 or13 you13 can13 switch13 to13 relave13 or13 absolute13 me13 ranges13

Quick13 me13 ranges13 allow13 switching13 between13 last13 X13 minutes13 to13 last13 Y13 hours13 or13 Z13 days13 etc13

Relave13 me13 ranges13 allow13 more13 specific13 versions13 of13 the13 Quick13 opons13 from13 specific13 minuteshoursdays13 to13 now13

13

Absolute13 me13 ranges13 specify13 the13 start13 and13 end13 mes13 for13 the13 searches13

13

Custom13 search13 queries13 You13 can13 use13 the13 text13 window13 of13 the13 Discover13 page13 to13 search13 for13 matches13 within13 the13 log13 records13 You13 can13 use13 Apache13 Lucene13 or13 Elascsearch13 query13 DSL13 syntax13

13

Page of 23 71 Page of 23 71

Saved13 searches13 You13 can13 save13 any13 custom13 search13 syntax13 and13 choice13 of13 me13 range13 for13 future13 ease13 of13 use13 Any13 relave13 or13 absolute13 me13 frame13 is13 saved13 as13 part13 of13 the13 search13 13

To13 save13 a13 search13

1 Set13 up13 and13 run13 the13 required13 syntax13 and13 me13 frame13

13

2 Click13 the13 Save13 Search13 icon13

13

3 Type13 in13 a13 name13 for13 the13 saved13 search13

Page of 24 71 Page of 24 71

13

7 To13 delete13 or13 update13 saved13 searches13 click13 13 manage13 saved13 searches13

13

Use13 the13 edit13 icons13 to13 make13 any13 changes13

Page of 26 71 Page of 26 71

13

Build13 simple13 Kibana13 histograms13 Kibana13 allows13 you13 to13 build13 charts13 and13 dashboards13 based13 on13 the13 search13 capability13 offered13 by13 the13 Discover13 page13 Saved13 searches13 can13 be13 used13 as13 the13 basis13 for13 charts13 and13 dashboards13 built13 using13 those13 charts13 To13 build13 a13 simple13 histogram13

1 Select13 the13 Visualize13 page13

13

2 Select13 Ver0cal13 Bar13 Chart13

13

3 To13 use13 a13 saved13 search13 click13 From13 a13 saved13 search13

Page of 27 71 Page of 27 71

13

4 Select13 a13 previously13 saved13 search13

13

5 Select13 to13 edit13 the13 x-shy‐axis

6 Edit13 the13 fields13 with13 informaon13 about13 the13 aggregaon13 field13 interval13 and13 custom13 label13

Page of 28 71 Page of 28 71

13

Page of 29 71 Page of 29 71

7 Click13 the13 green13 arrow13 The13 histogram13 is13 created13 and13 run13 based13 on13 the13 saved13 search

13

8 To13 save13 the13 histogram13 click13 the13 Save13 Visualiza0on13 icon13 and13 add13 a13 tle13

13

13

9 Click13 Save13

10 To13 view13 the13 visualisaon13 click13 the13 Dashboard13 page13

Page of 30 71 Page of 30 71

13

11 Click13 the13 Add13 Visualiza0on13 icon13

13

12 Select13 the13 visiualizaon13 that13 you13 just13 created13

13

Page of 31 71 Page of 31 71

13

13 To13 save13 the13 dashboard13 click13 the13 Save13 Dashboard13 icon13

Page of 32 71 Page of 32 71

13

14 Add13 a13 tle13 and13 select13 whether13 to13 store13 the13 selected13 me13 frame13 with13 the13 dashboard13

13

15 Click13 Save13

16 You13 can13 edit13 the13 shape13 and13 number13 of13 charts13 and13 save13 the13 dashboard13

17 To13 re-shy‐run13 the13 dashboard13 select13 the13 Dashboard13 page13 and13 click13 the13 Load13 Saved13 Dashboard13 icon13

Page of 33 71 Page of 33 71

13

18 Select13 the13 saved13 dashboard13 from13 the13 list13

13

19 The13 dashboard13 runs13

13

Page of 34 71 Page of 34 71

Page of 35 71 Page of 35 71

Aler0ng13 with13 ELK13 ELK13 examines13 the13 log13 records13 in13 process13 to13 send13 alerts13 to13 third-shy‐party13 alerng13 event13 management13 and13 social13 media13 applicaons13 As13 shown13 previously13 in13 the13 ELK13 architecture13 the13 ELK13 strategy13 for13 alerng13 is13 to13 send13 alerts13 from13 the13 Logstash13 component13 directly13 Therefore13 log13 records13 and13 alerts13 are13 sent13 to13 both13 Elascsearch13 and13 third-shy‐party13 tools13 at13 the13 same13 instant13 using13 the13 output13 secon13 of13 the13 Logstash13 configuraon13 Any13 alerts13 are13 generated13 during13 the13 filter13 secon13 processing13 of13 the13 Logstash13 configuraon13

Create13 alerts13 in13 Logstash13 The13 Logstash13 filter13 secon13 is13 used13 to13 trigger13 alerts13 based13 on13 the13 log13 records13 being13 processed13 For13 example13 log13 records13 can13 be13 checked13 for13 errors13 warnings13 crashed13 or13 stopped13 messages13 and13 generate13 an13 alert13

Here13 is13 an13 example13 Logstash13 filter13 code13

13 if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

The13 specific13 tags13 are13 set13 against13 the13 log13 records13 to13 trigger13 the13 alert13 later13 in13 the13 configuraon13 in13 the13 output13 secon13

Control13 the13 number13 of13 generated13 alerts13 using13 the13 Logstash13 throcle13 plugin13 so13 that13 either13 a13 lot13 of13 instances13 in13 a13 fixed13 me13 period13 are13 required13 before13 an13 alert13 is13 sent13 or13 a13 specific13 number13 of13 instances13 triggers13 the13 alert13

Page of 36 71 Page of 36 71

Edit13 the13 plugin13 parameters13 to13 meet13 system13 requirements13

13 13 13 13 13 if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 10

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle13

after_count =gt 20

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 30

period =gt 1200

key =gt servicename_alarm_warning

Page of 37 71 Page of 37 71

add_tag =gt servicename_throttled

13 13 13 13 13 13 13 13 13

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 40

period =gt 1500

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

Enrich13 alerts13 with13 message13 detail13 Some13 of13 the13 third-shy‐party13 integraons13 benefit13 from13 having13 components13 of13 the13 raw13 log13 record13 sent13 in13 addion13 to13 the13 raw13 record13 or13 fields13 in13 the13 record13 edited13 transformed13 or13 translated13

Grok13 Logstash13 allows13 the13 log13 record13 to13 be13 broken13 up13 into13 logical13 secons13 and13 each13 secon13 stored13 in13 a13 new13 parameter13 using13 the13 grok13 parameter13 in13 the13 filter13 secon13 Here13 is13 an13 example13 grok13

13 grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

To13 build13 the13 grok13 statement13 Logstash13 offers13 an13 online13 tool13 at13 hcpsgrokdebugherokuappcom13

Translate13 If13 specific13 fields13 in13 the13 log13 record13 are13 coded13 (for13 example13 the13 Bluemix13 app13 name13 is13 oaen13 not13 sent13 in13 the13 syslog13 messages13 instead13 the13 GUID13 is13 sent13 which13 is13 not13 useful)13 then13 the13 filter13 secon13 translate13 plugin13 can13 be13 used13 to13 add13 new13 fields13 by13 holding13 the13 decoded13 versions13 of13 those13 values13

13 translate

field =gt data1

destination =gt data1

override =gt true

Page of 38 71 Page of 38 71

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

dc50f6e4-c052-436a-bf27-95ba392c4228 monverifyservicemanagement

0c677e82-5bfc-4e38-9155-e416249e3ee6 ms-catalog-postpyretic-pomiculture

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

However13 the13 translate13 plugin13 is13 not13 included13 with13 the13 Logstash13 installaon13 In13 the13 next13 secon13 I13 will13 13 describe13 how13 to13 add13 new13 plugins13 to13 the13 Logstash13 instance13

Add13 custom13 plugins13 to13 Logstash13 As13 an13 open13 source13 tool13 Logstash13 encourages13 users13 to13 develop13 content13 for13 all13 users13 to13 share13 Some13 of13 the13 plugins13 that13 were13 created13 by13 our13 community13 include13 a13 translaon13 plugin13 an13 output13 Slack13 plugin13 and13 an13 output13 syslog13 plugin13 Letrsquos13 look13 at13 how13 to13 add13 these13 plugins13 to13 your13 environment13

Filter13 translate13 plugin13 To13 add13 the13 translate13 plugin13

1 Download13 the13 plugin13

2 Copy13 the13 plugin13 file13 logstash-shy‐filter-shy‐translate-shy‐214gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-filter-translate 214

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Output13 Slack13 plugin13 To13 add13 the13 Slack13 plugin13

Page of 39 71 Page of 39 71

1 Download13 the13 plugin13 13

2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐slack-shy‐014gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-output-slack 014

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Output13 syslog13 plugin13 To13 add13 the13 syslog13 plugin13

1 Download13 the13 plugin13 13

2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐syslog-shy‐214gem13 to13 the13 ELK13 Virtual13 Server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-output-syslog 214

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Integrate13 Logstash13 with13 Slack13 To13 send13 generated13 alerts13 to13 Slack13 complete13 these13 steps13

1 Create13 a13 Slack13 integraon13 13

2 Use13 the13 Slack13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 Slack13 integraon13

113 Login13 to13 Slack13 and13 go13 to13 the13 Apps13 page13 13

Page of 40 71 Page of 40 71

13

313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13

13

A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13

13

Page of 42 71 Page of 42 71

The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13

13

The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13

13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

slack

url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5

channel =gt cloud-operations-mon

username =gt nick_cawood

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13

Page of 43 71 Page of 43 71

Page of 44 71 Page of 44 71

In13 Slack13 the13 alerts13 look13 like13 this13

13

Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13

1 Create13 a13 PagerDuty13 integraon13 13

2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13

1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13

13

2 Click13 Add13 New13 Service13

13

313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13

Page of 45 71 Page of 45 71

13

413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13

513 Click13 Add13 Service13

13

The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13

Page of 46 71 Page of 46 71

13

The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt be7eb1ea9c224018923e818d3e891d0b

incident_key =gt logstashhosttype

Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13

Page of 47 71 Page of 47 71

13

Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13

1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13

2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13

3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13

4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13

13 13 http

url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo

http_method =gt post

format =gt json

For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13

13 13 syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

Page of 48 71 Page of 48 71

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13

13

The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13

if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2

This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13

Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13

Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13

Logstash13 severity Shown13 in13 Omnibus13 as

alert Indeterminate

crical Warning

debug 7

emergency Clear

error Minor

informaonal 6

noce Crical

warning Major

Page of 49 71 Page of 49 71

13

Page of 50 71 Page of 50 71

Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13

input

tcp

port =gt 5000

type =gt syslog

filter

mutate

add_field =gt format =gt cf

grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

translate

field =gt data1

destination =gt data1

override =gt true

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

=== Check Alarm Criteria ===

Standard Key Words

if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

=== Alarm Exclusions ===

if [message] =~ (i)sometexthere

mutate remove_tag =gt servicename_alarm

Throttle alarms so we do not hit PD rate limits

Page of 52 71 Page of 52 71

if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 5

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle

after_count =gt 5

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 5

period =gt 1200

key =gt servicename_alarm_warning

add_tag =gt servicename_throttled

Page of 53 71 Page of 53 71

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

output

elasticsearch

hosts =gt localhost

stdout codec =gt rubydebug

if servicename_alarm_err in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

Page of 54 71 Page of 54 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 55 71 Page of 55 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

Page of 56 71 Page of 56 71

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert err message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_error in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

Page of 57 71 Page of 57 71

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 58 71 Page of 58 71

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 59 71 Page of 59 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert error message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

Page of 60 71 Page of 60 71

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

Page of 61 71 Page of 61 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 62 71 Page of 62 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt x

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert fatal message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

Page of 63 71 Page of 63 71

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

Page of 64 71 Page of 64 71

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 65 71 Page of 65 71

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt heavy_exclamation_mark

Page of 66 71 Page of 66 71

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert WARNING message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 67 71 Page of 67 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

Page of 68 71 Page of 68 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 69 71 Page of 69 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

Page of 70 71 Page of 70 71

Page of 71 71 Page of 71 71

Page 7: How to integrate the Elasticsearch Logstash Kibana (ELK ... · How to integrate the Elasticsearch Logstash Kibana (ELK) log analytics stack into IBM Bluemix Nick Cawood IBM Cloud

13

The13 Bluemix13 cloud13 plaRorm13 offers13 a13 Public13 network13 and13 access13 to13 US13 South13 and13 UK13 environments13 From13 all13 these13 environments13 third-shy‐party13 cloud-shy‐based13 soluons13 are13 also13 accessible13

The13 ELK13 stack13 is13 installed13 in13 the13 Bluemix13 United13 Kingdom13 environment13 because13 this13 is13 the13 environment13 where13 Bluemix13 has13 virtual13 servers13 available13 for13 provisioning13 Other13 applicaons13 in13 the13 CASE13 project13 were13 also13 installed13 on13 virtual13 servers13 in13 the13 Bluemix13 United13 Kingdom13 region13

Bluemix13 apps13 were13 used13 to13 create13 a13 system13 to13 be13 monitored13 by13 ELK13 These13 apps13 were13 hosted13 in13 the13 Bluemix13 US13 South13 environment13 using13 Bluemix13 microservices13 which13 produce13 logs13 during13 normal13 operaon13 and13 during13 crashes13 and13 outages13

Through13 the13 Bluemix13 Cloud13 Foundry13 process13 the13 log13 records13 can13 be13 broadcast13 from13 the13 Bluemix13 apps13 so13 Logstash13 can13 receive13 them13 and13 send13 them13 to13 Elascsearch13 within13 the13 ELK13 stack13 or13 other13 virtual13 servers13 (hosng13 other13 IBM13 or13 third-shy‐party13 products)13 in13 the13 Bluemix13 United13 Kingdom13 environment13 or13 to13 third-shy‐party13 applicaons13 in13 the13 public13 cloud13 (Pager13 Duty13 Slack13 etc)13

Crea0ng13 a13 virtual13 server13 in13 Bluemix13 With13 Bluemix13 you13 can13 provision13 a13 virtual13 server13 in13 the13 cloud13 to13 install13 standalone13 or13 integrated13 system13 components13 Not13 all13 Bluemix13 regions13 offer13 this13 service13 all13 the13 me13 due13 to13 beta-shy‐project13 constraints13 In13 this13 arclersquos13 examples13 virtual13 servers13 were13 allowed13 to13 be13 provisioned13 in13 the13 United13 Kingdom13 region13 For13 more13 informaon13 read13 how13 to13 provision13 a13 Bluemix13 virtual13 server13 including13 some13 simple13 setup13 steps13 for13 ease13 of13 use13

Page of 7 71 Page of 7 71

Create13 an13 SSH13 key13 To13 access13 your13 virtual13 server13 once13 it13 is13 provisioned13 allocate13 an13 SSH13 key13 to13 the13 virtual13 server13

To13 generate13 a13 publicprivate13 RSA13 key13 pair13 follow13 these13 steps13

1 In13 a13 UNIX13 or13 Linuxreg13 window13 run13 the13 following13 command13 specifying13 a13 ltuniquekeynamegt13 of13 your13 your13 choice13 and13 a13 password13 or13 passphrase13 (memorize13 the13 password)$13 ssh-shy‐keygen13 -shy‐t13 rsa13 -shy‐f13 ltuniquekeynamegt13

2 Enter13 the13 passphrase13 (empty13 for13 no13 passphrase)13

3 Enter13 the13 same13 passphrase13 again13

Your13 idenficaon13 has13 been13 saved13 in13 ltuniquekeynamegt13

Your13 public13 key13 has13 been13 saved13 in13 ltuniquekeynamegtpub13

The13 key13 fingerprint13 is13

d6268464506525930ce4632c693c203c13 ltusergtltservergt13

The13 keys13 randomart13 image13 is13

+--[ RSA 2048]----+

|o ++=o |

| E o ooo |

| |

| + o |

| S o |

| o |

| |

| |

| |

+-----------------+

Two13 keys13 are13 created13 public13 and13 private13 The13 public13 key13 contents13 are13 used13 to13 create13 a13 virtual13 server13 with13 a13 public13 IP13 address13 The13 private13 key13 is13 used13 to13 access13 the13 virtual13 server13 from13 a13 client13 machine13

Page of 8 71 Page of 8 71

Provisioning13 a13 virtual13 server13 1 A13 Bluemix13 account13 is13 required13 for13 this13 process13 Sign13 up13 here13 and13 log13 in13 Make13 sure13 your13

environment13 is13 correct13 For13 example13 if13 you13 are13 in13 the13 United13 Kingdom13 select13 the13 United13 Kingdom13 region13 in13 the13 Bluemix13 console13

13

13

2 Add13 a13 virtual13 server13 to13 your13 Bluemix13 environment13 Click13 Run13 Virtual13 Servers13

13

3 Select13 an13 OS13 image13 from13 the13 EXISTING13 list13 For13 ELK13 select13 Centos13 OS13 (use13 latest13 available13 version)13

Page of 9 71 Page of 9 71

4 Name13 the13 virtual13 server13

5 Make13 sure13 Assign13 public13 IP13 address13 is13 checked13

13

6 Click13 Add13 Key13 to13 add13 the13 SSH13 key13 you13 created13 previously13 Copy13 the13 contents13 of13 the13 public13 key13 into13 the13 Public13 Key13 to13 import13 field13

Page of 10 71 Page of 10 71

13

7 Click13 OK13 Then13 click13 CREATE13

The13 virtual13 server13 is13 provisioned13 and13 added13 to13 the13 Bluemix13 dashboard13 under13 VIRTUAL13 SERVERS13

Page of 11 71 Page of 11 71

13

Set13 up13 the13 virtual13 server13

Log13 on13 to13 the13 virtual13 server13 Now13 you13 can13 log13 on13 to13 the13 virtual13 server13 and13 add13 new13 users13 and13 programs13 to13 make13 it13 easier13 to13 access13 the13 virtual13 server13

To13 access13 the13 server13 go13 to13 the13 UNIX13 or13 Linux13 window13 that13 you13 used13 to13 create13 the13 SSH13 key13 In13 the13 directory13 where13 the13 ltuniquekeynamegt13 files13 exist13 run13 the13 following13 command13 referencing13 the13 SSH13 keyname13 the13 public13 IP13 address13 Bluemix13 allocated13 for13 the13 virtual13 server13 and13 the13 password13 for13 the13 SSH13 key13

[userserver ~]$ ssh -i ltuniquekeynamegt ibmcloudltpublicIPaddressgt

Enter13 the13 passphrase13 for13 the13 key13 ltuniquekeynamegt13

Last login Tue Jul 5 125546 2016 from 108-210-96-231lightspeedrlghncsbcglobalnet

[ibmcloudelkforcase ~]$

ibmcloud13 is13 the13 default13 user13 for13 the13 Bluemix13 virtual13 servers13 and13 allows13 SSH13 to13 access13 the13 server13

Acces13 the13 root13 user13 To13 access13 the13 root13 user13 log13 on13 as13 ibmcloud13 user13 and13 run13 the13 following13 command13

[ibmcloudelkforcase ~]$ sudo bash

[rootelkforcase ibmcloud]

The13 root13 user13 can13 be13 used13 to13 add13 programs13 to13 the13 server13 and13 create13 and13 edit13 users13 as13 required13 to13 access13 applicaons13 installed13 on13 the13 virtual13 server13

Page of 12 71 Page of 12 71

Copy13 files13 to13 the13 virtual13 server13 To13 move13 installaon13 files13 to13 the13 server13 go13 to13 the13 UNIX13 or13 Linux13 window13 you13 used13 to13 create13 the13 SSH13 key13 and13 the13 directory13 where13 the13 ltuniquekeynamegt13 files13 exist13 Move13 the13 installaon13 files13 to13 the13 same13 locaon13

Run13 the13 following13 command13 referencing13 the13 SSH13 keyname13 the13 public13 IP13 address13 Bluemix13 allocated13 for13 the13 virtual13 server13 and13 the13 password13 for13 the13 SSH13 key13

[userserver ~]$ scp ndashi ltuniquekeynamegt ltINSTALLATIONFILENAMEgt ibmcloudltpublicIPaddressgthomeibmcloud

Enter13 the13 passphrase13 for13 key13 cloudkey313

This13 copies13 the13 file13 using13 the13 ibmcloud13 user13 to13 the13 homeibmcloud13 directory13 on13 the13 virtual13 server13

Install13 and13 access13 the13 virtual13 server13 via13 VNC13 server13 Using13 the13 ssh13 command13 to13 access13 the13 virtual13 server13 is13 not13 the13 most13 stable13 method13 and13 does13 not13 offer13 X-shy‐Windows13 support13 Therefore13 we13 suggest13 that13 you13 use13 a13 VNC13 client13 to13 access13 the13 virtual13 server13

Installing13 a13 VNC13 client13 should13 be13 done13 as13 the13 root13 user13 Use13 the13 yum13 command13 as13 follows13

[ibmcloudelkforcase ~]$ yum groupinstall GNOME Desktop

[ibmcloudelkforcase ~]$ yum install tigervnc-server

These13 commands13 first13 install13 a13 desktop13 style13 OS13 user13 interface13 and13 then13 a13 VNC13 client13 to13 remotely13 access13 the13 desktop13 style13 OS13

Then13 as13 either13 the13 root13 user13 or13 another13 user13 (example13 below13 uses13 elk)13 run13 the13 following13 command13 to13 create13 the13 VNC13 client13 connecon13

[elkelkforcase ~]$ vncserver

New elkforcase2 (elk) desktop is elkforcase2

Starting applications specified in homeelkvncxstartup

Log file is homeelkvncelkforcase2log

[elkelkforcase ~]$

Install13 a13 VNC13 client13 on13 the13 client13 machine13 used13 to13 access13 the13 Internet13 and13 (in13 the13 example13 above)13 use13 the13 command13 ltpublicIPadddressgt213 and13 the13 password13 for13 ELK13 to13 access13 the13 virtual13 server13

Using13 a13 VNC13 client13 means13 that13 you13 can13 leave13 long13 duraon13 installaons13 running13 without13 fear13 of13 the13 terminal13 losing13 connecon13 and13 the13 installaons13 failing13

Install13 the13 ELK13 stack13 Download13 the13 source13 soaware13 code13 for13 the13 ELK13 stack13 from13 the13 ELK13 website13 13

At13 the13 me13 of13 wring13 the13 downloaded13 soaware13 files13 were13

Page of 13 71 Page of 13 71

13

Transfer13 the13 files13 to13 the13 virtual13 server13 (using13 the13 scp13 method13 detailed13 above)13

Install13 Elas0csearch13 As13 a13 root13 user13 create13 a13 non-shy‐root13 user13 with13 the13 adduser13 command13

Create13 a13 directory13 to13 install13 Elascsearch13 into13 and13 untar13 the13 contents13 of13 the13 Elascsearch13 install13 file13 into13 that13 directory13

[elkelkforcase]$ cd home

[elkelkforcase]$ mkdir elasticsearch

[elkelkforcase]$ cp homeibmcloudelasticsearch-233targz homeelasticsearch

[elkelkforcase]$ cd elasticsearch

[elkelkforcase]$ gunzip elasticsearch-233targz

[elkelkforcase]$ untar elasticsearch-233tar

Elascsearch13 is13 now13 installed13

Start13 Elas0csearch13 To13 start13 Elascsearch13 go13 to13 the13 bin13 directory13 as13 the13 non-shy‐root13 user13 when13 you13 installed13 it13 and13 run13 the13 elascsearch13 executable13

[elkelkforcase]$ cd homeelasticsearchelasticsearch-233bin

[elkelkforcase]$ elasticsearch amp

[2] 22211

[elkelkforcase]$ [2016-07-05 133857730][INFO ][node] [Arsenal] version[233] pid[22211] build[218bdf12016-05-17T154004Z]

[2016-07-05 133857740][INFO ][node] [Arsenal] initializing

[2016-07-05 133858984][INFO ][plugins] [Arsenal] modules [reindex lang-expression lang-groovy] plugins [] sites []

[2016-07-05 133859146][INFO ][env] [Arsenal] using [1] data paths mounts [[ (rootfs)]] net usable_space [155gb] net total_space [199gb] spins [unknown] types [rootfs]

[2016-07-05 133859146][INFO ][env] [Arsenal] heap size [10156mb] compressed ordinary object pointers [true]

Page of 14 71 Page of 14 71

[2016-07-05 133859147][WARN ][env] [Arsenal] max file descriptors [4096] for elasticsearch process likely too low consider increasing to at least [65536]

[2016-07-05 133903526][INFO ][node] [Arsenal] initialized

[2016-07-05 133903526][INFO ][node] [Arsenal] starting

To13 test13 that13 Elascsearch13 is13 running13 correctly13 open13 the13 browser13 in13 the13 VNC13 client13 and13 run13 localhost920013

13

Install13 Kibana13 ELK13 recommends13 installing13 and13 running13 Kibana13 as13 root13 13

1 As13 root13 user13 create13 an13 installaon13 directory13 and13 untar13 the13 installaon13 file13

[rootelkforcase]$ cd home

[rootelkforcase]$ mkdir kibana

[rootelkforcase]$ cp homeibmcloudkibana-451-linux-x64targz homekibana

[rootelkforcase]$ cd kibana

[rootelkforcase]$ gunzip kibana-451-linux-x64targz

[rootelkforcase]$ untar kibana-451-linux-x64tar

2 Modify13 the13 manifestyml13 configuraon13 file13 Go13 to13 the13 kibana-shy‐451-shy‐linux-shy‐x6413 directory13

[rootelkforcase]$ cd homekibanakibana-451-linux-x64

3 Edit13 the13 manifestyml13 file13 so13 the13 contents13 are13

applicaons13

-shy‐ name13 kibana-shy‐in-shy‐bluemix13

memory13 128M13

host13 kibana-shy‐in-shy‐bluemix13

buildpack13 hcpsgithubcomcloudfoundry-shy‐communitynginx-shy‐buildpackgit13

Page of 15 71 Page of 15 71

4 Edit13 the13 configkibanayml13 file13 so13 the13 contents13 are13

Kibana is served by a back end server This controls which port to use

port 5601

The host to bind the server to

host 0000

If you are running kibana behind a proxy and want to mount it at a path

specify that path here The basePath cant end in a slash

serverbasePath

The maximum payload size in bytes on incoming server requests

servermaxPayloadBytes 1048576

The Elasticsearch instance to use for all your queries

elasticsearchurl httplocalhost9200

preserve_elasticsearch_host true will send the hostname specified in `elasticsearch` If you set it to false

then the host you use to connect to this Kibana instance will be sent

elasticsearchpreserveHost true

Kibana uses an index in Elasticsearch to store saved searches visualizations

and dashboards It will create a new index if it doesnt already exist

kibanaindex kibana

The default application to load

kibanadefaultAppId discover

If your Elasticsearch is protected with basic auth these are the user credentials

used by the Kibana server to perform maintenance on the kibana_index at startup Your

Kibana users will still need to authenticate with Elasticsearch (which is proxied through

the Kibana server)

Page of 16 71 Page of 16 71

elasticsearchusername user

elasticsearchpassword pass

SSL for outgoing requests from the Kibana Server to the browser (PEM formatted)

serversslcert pathtoyourservercrt

serversslkey pathtoyourserverkey

Optional setting to validate that your Elasticsearch backend uses the same key files (PEM formatted)

elasticsearchsslcert pathtoyourclientcrt

elasticsearchsslkey pathtoyourclientkey

If you need to provide a CA certificate for your Elasticsearch instance put

the path of the pem file here

elasticsearchsslca pathtoyourCApem

Set to false to have a complete disregard for the validity of the SSL

certificate

elasticsearchsslverify true

Time in milliseconds to wait for elasticsearch to respond to pings defaults to

request_timeout setting

elasticsearchpingTimeout 1500

Time in milliseconds to wait for responses from the back end or elasticsearch

This must be gt 0

elasticsearchrequestTimeout 30000

Time in milliseconds for Elasticsearch to wait for responses from shards

Set to 0 to disable

elasticsearchshardTimeout 0

Time in milliseconds to wait for Elasticsearch at Kibana startup before retrying

elasticsearchstartupTimeout 5000

elasticsearchstartupTimeout 30000

Page of 17 71 Page of 17 71

Set the path to where you would like the process id file to be created

pidfile varrunkibanapid

If you would like to send the log output to a file you can set the path below

loggingdest stdout

Set this to true to suppress all logging output

loggingsilent false

Set this to true to suppress all logging output except for error messages

loggingquiet false

Set this to true to log all events including system usage information and all requests

loggingverbose false

Kibana13 is13 now13 installed13

Start13 Kibana13 To13 start13 Kibana13 go13 to13 the13 bin13 directory13 and13 run13 the13 kibana13 executable13

[rootelkforcase]$

[rootelkforcase]$ cd homekibanakibana-451-linux-x64bin

[rootelkforcase]$ kibana amp

[1] 22283

[rootelkforcase] log [095900542] [info][status][pluginkibana] Status changed from uninitialized to green - Ready

log [095900600] [info][status][pluginelasticsearch] Status changed from uninitialized to yellow - Waiting for Elasticsearch

log [095900689] [info][status][pluginkbn_vislib_vis_types] Status changed from uninitialized to green - Ready

log [095900717] [info][status][pluginmarkdown_vis] Status changed from uninitialized to green - Ready

log [095900721] [info][status][pluginmetric_vis] Status changed from uninitialized to green - Ready

log [095900814] [info][status][pluginspyModes] Status changed from uninitialized to green - Ready

log [095900934] [info][status][pluginstatusPage] Status changed from uninitialized to green - Ready

Page of 18 71 Page of 18 71

log [095900941] [info][status][plugintable_vis] Status changed from uninitialized to green - Ready

log [095901067] [info][listening] Server running at http00005601

log [095906083] [info][status][pluginelasticsearch] Status changed from yellow to yellow - No existing Kibana index found

log [095908942] [info][status][pluginelasticsearch] Status changed from yellow to green - Kibana index ready

Kibana13 is13 now13 started13

Broadcast13 Bluemix13 app13 log13 records13 Bluemix13 apps13 (which13 are13 shown13 in13 the13 US13 South13 Bluemix13 region13 of13 the13 System13 Architecture13 diagram)13 will13 produce13 logs13 in13 a13 syslog13 rfc542413 format13 using13 a13 Bluemix13 tool13 called13 Loggregator13

For13 Logstash13 to13 receive13 these13 logs13 each13 Bluemix13 app13 needs13 to13 broadcast13 these13 logs13 To13 do13 that13 run13 the13 following13 commands13 on13 each13 Bluemix13 app13 using13 the13 public13 IP13 address13 of13 the13 ELK13 virtual13 server13 the13 ELK13 tag13 to13 trace13 the13 process13 and13 insert13 the13 name13 of13 each13 app13 in13 turn13

cf cups elk -l syslogltpublicIPaddress5000 cf bind-service ltmyappgt elk

cf restage ltmyappgt

Install13 Logstash13 To13 install13 Logstash13 as13 root13 user13 create13 a13 directory13 for13 the13 installaon13 and13 untar13 the13 installaon13 file13 there13

13 13 [rootelkforcase]$ cd home

[rootelkforcase]$ mkdir logstash

[rootelkforcase]$ cp homeibmcloudlogstash-233targz homelogstash

[rootelkforcase]$ cd logstash

[rootelkforcase]$ gunzip logstash-233targz

[rootelkforcase]$ untar logstash-233tar

Write13 Logstash13 configura0on13 Logstash13 requires13 a13 configuraon13 file13 to13 be13 wricen13 to13 receive13 transform13 and13 output13 the13 log13 records13 to13 Elascsearch13 This13 secon13 explains13 the13 basic13 requirements13 of13 processing13 Bluemix13 app13 log13 records13 into13 Elascsearch13 (more13 complex13 configuraons13 are13 detailed13 later13 in13 this13 document13 for13 the13 integraons13 with13 third-shy‐party13 tools)13

Create13 a13 conf13 file13 as13 root13 user13

[rootelkforcase]$ cd homelogstashlogstash-233

[rootelkforcase]$ mkdir config

Page of 19 71 Page of 19 71

[rootelkforcase]$ cd config

[rootelkforcase]$ touch loggregatorconf

Edit13 the13 loggregator13 file13 with13 the13 following13 contents13

13 Input13 secon13 defines13 which13 log13 records13 to13 receive13

13 port13 and13 type13 reflect13 method13 selected13 for13 Cloud13 Foundry13 to13 broadcast13 the13 records13

input

tcp

port =gt 5000

type =gt syslog

13 Filter13 secon13 ndash13 transforms13 the13 data13

filter

Add a field to identify the records as Cloud Foundry based

mutate

add_field =gt format =gt cf

13 Output13 secon13 ndash13 sends13 the13 data13 to13 its13 desnaon(s)13

output

13 Use13 the13 elascsearch13 plugin13 to13 send13 the13 data13 to13 the13 elascsearch13 instance13

13 elasticsearch

hosts =gt localhost

stdout codec =gt rubydebug

13 End13 of13 Config13

Save13 the13 file13 Verify13 the13 format13 of13 the13 file13 is13 acceptable13 by13 running13 the13 following13 commands13

[rootelkforcase]$ cd homelogstashlogstash-233

[rootelkforcase]$ binlogstash agent --configtest -f configloggregatorconf

Page of 20 71 Page of 20 71

Configuraon13 is13 OK13

Start13 Logstash13 Aaer13 you13 have13 wricen13 and13 verified13 the13 Logstash13 configuraon13 you13 can13 start13 Logstash13 to13 process13 records13 from13 the13 Bluemix13 apps13 (via13 Cloud13 Foundry)13 Run13 the13 following13 as13 root13 user13

[rootelkforcase]$ cd homelogstashlogstash-233

[rootelkforcase]$ binlogstash agent -f configloggregatorconf

Logstash13 is13 now13 started13

Monitor13 Logstash13 processing13 Once13 Logstash13 is13 running13 and13 log13 records13 are13 being13 produced13 by13 the13 Bluemix13 apps13 you13 can13 monitor13 progress13 in13 the13 same13 window13 that13 was13 used13 to13 start13 Logstash13 Output13 is13 similar13 to13 the13 following13

13

The13 log13 records13 being13 broadcast13 from13 the13 Bluemix13 apps13 are13 now13 sent13 to13 Elascsearch13 from13 the13 Logstash13 output13 configuraon13 Records13 are13 sent13 in13 real13 me13 If13 Logstash13 is13 not13 running13 while13 log13 records13 are13 being13 produced13 then13 these13 records13 will13 not13 be13 processed13 by13 Logstash13 and13 will13 not13 be13 stored13 in13 Elascsearch13

Page of 21 71 Page of 21 71

Use13 ELK13 The13 log13 records13 from13 the13 Bluemix13 apps13 are13 now13 being13 stored13 and13 indexed13 by13 the13 Elascsearch13 component13 This13 data13 can13 be13 accessed13 via13 the13 Kibana13 applicaon13

Kibana13 Discover13 page13 Using13 a13 web13 browser13 either13 directly13 from13 a13 client13 or13 using13 the13 VNC13 client13 access13 Kibana13 using13 the13 publicIPaddress13 of13 the13 ELK13 virtual13 server13

hcpltpublicIPaddressgt560113

No13 userpassword13 credenals13 are13 required13

Kibana13 defaults13 to13 a13 search13 of13 all13 records13 ()13 in13 the13 last13 1513 minutes13 and13 returns13 the13 results13

13

More13 complex13 searches13 are13 possible13 by13 changing13 the13 me13 filter13 and13 supplying13 search13 queries13

Change13 the13 0me13 filter13 To13 change13 the13 me13 ranges13 click13 the13 clock13 symbol13 in13 the13 top-shy‐right13 of13 the13 Kibana13 Discover13 page13

13

You13 will13 see13 the13 following13 opons13

13

Page of 22 71 Page of 22 71

A13 variety13 of13 quick13 me13 ranges13 can13 be13 selected13 or13 you13 can13 switch13 to13 relave13 or13 absolute13 me13 ranges13

Quick13 me13 ranges13 allow13 switching13 between13 last13 X13 minutes13 to13 last13 Y13 hours13 or13 Z13 days13 etc13

Relave13 me13 ranges13 allow13 more13 specific13 versions13 of13 the13 Quick13 opons13 from13 specific13 minuteshoursdays13 to13 now13

13

Absolute13 me13 ranges13 specify13 the13 start13 and13 end13 mes13 for13 the13 searches13

13

Custom13 search13 queries13 You13 can13 use13 the13 text13 window13 of13 the13 Discover13 page13 to13 search13 for13 matches13 within13 the13 log13 records13 You13 can13 use13 Apache13 Lucene13 or13 Elascsearch13 query13 DSL13 syntax13

13

Page of 23 71 Page of 23 71

Saved13 searches13 You13 can13 save13 any13 custom13 search13 syntax13 and13 choice13 of13 me13 range13 for13 future13 ease13 of13 use13 Any13 relave13 or13 absolute13 me13 frame13 is13 saved13 as13 part13 of13 the13 search13 13

To13 save13 a13 search13

1 Set13 up13 and13 run13 the13 required13 syntax13 and13 me13 frame13

13

2 Click13 the13 Save13 Search13 icon13

13

3 Type13 in13 a13 name13 for13 the13 saved13 search13

Page of 24 71 Page of 24 71

13

7 To13 delete13 or13 update13 saved13 searches13 click13 13 manage13 saved13 searches13

13

Use13 the13 edit13 icons13 to13 make13 any13 changes13

Page of 26 71 Page of 26 71

13

Build13 simple13 Kibana13 histograms13 Kibana13 allows13 you13 to13 build13 charts13 and13 dashboards13 based13 on13 the13 search13 capability13 offered13 by13 the13 Discover13 page13 Saved13 searches13 can13 be13 used13 as13 the13 basis13 for13 charts13 and13 dashboards13 built13 using13 those13 charts13 To13 build13 a13 simple13 histogram13

1 Select13 the13 Visualize13 page13

13

2 Select13 Ver0cal13 Bar13 Chart13

13

3 To13 use13 a13 saved13 search13 click13 From13 a13 saved13 search13

Page of 27 71 Page of 27 71

13

4 Select13 a13 previously13 saved13 search13

13

5 Select13 to13 edit13 the13 x-shy‐axis

6 Edit13 the13 fields13 with13 informaon13 about13 the13 aggregaon13 field13 interval13 and13 custom13 label13

Page of 28 71 Page of 28 71

13

Page of 29 71 Page of 29 71

7 Click13 the13 green13 arrow13 The13 histogram13 is13 created13 and13 run13 based13 on13 the13 saved13 search

13

8 To13 save13 the13 histogram13 click13 the13 Save13 Visualiza0on13 icon13 and13 add13 a13 tle13

13

13

9 Click13 Save13

10 To13 view13 the13 visualisaon13 click13 the13 Dashboard13 page13

Page of 30 71 Page of 30 71

13

11 Click13 the13 Add13 Visualiza0on13 icon13

13

12 Select13 the13 visiualizaon13 that13 you13 just13 created13

13

Page of 31 71 Page of 31 71

13

13 To13 save13 the13 dashboard13 click13 the13 Save13 Dashboard13 icon13

Page of 32 71 Page of 32 71

13

14 Add13 a13 tle13 and13 select13 whether13 to13 store13 the13 selected13 me13 frame13 with13 the13 dashboard13

13

15 Click13 Save13

16 You13 can13 edit13 the13 shape13 and13 number13 of13 charts13 and13 save13 the13 dashboard13

17 To13 re-shy‐run13 the13 dashboard13 select13 the13 Dashboard13 page13 and13 click13 the13 Load13 Saved13 Dashboard13 icon13

Page of 33 71 Page of 33 71

13

18 Select13 the13 saved13 dashboard13 from13 the13 list13

13

19 The13 dashboard13 runs13

13

Page of 34 71 Page of 34 71

Page of 35 71 Page of 35 71

Aler0ng13 with13 ELK13 ELK13 examines13 the13 log13 records13 in13 process13 to13 send13 alerts13 to13 third-shy‐party13 alerng13 event13 management13 and13 social13 media13 applicaons13 As13 shown13 previously13 in13 the13 ELK13 architecture13 the13 ELK13 strategy13 for13 alerng13 is13 to13 send13 alerts13 from13 the13 Logstash13 component13 directly13 Therefore13 log13 records13 and13 alerts13 are13 sent13 to13 both13 Elascsearch13 and13 third-shy‐party13 tools13 at13 the13 same13 instant13 using13 the13 output13 secon13 of13 the13 Logstash13 configuraon13 Any13 alerts13 are13 generated13 during13 the13 filter13 secon13 processing13 of13 the13 Logstash13 configuraon13

Create13 alerts13 in13 Logstash13 The13 Logstash13 filter13 secon13 is13 used13 to13 trigger13 alerts13 based13 on13 the13 log13 records13 being13 processed13 For13 example13 log13 records13 can13 be13 checked13 for13 errors13 warnings13 crashed13 or13 stopped13 messages13 and13 generate13 an13 alert13

Here13 is13 an13 example13 Logstash13 filter13 code13

13 if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

The13 specific13 tags13 are13 set13 against13 the13 log13 records13 to13 trigger13 the13 alert13 later13 in13 the13 configuraon13 in13 the13 output13 secon13

Control13 the13 number13 of13 generated13 alerts13 using13 the13 Logstash13 throcle13 plugin13 so13 that13 either13 a13 lot13 of13 instances13 in13 a13 fixed13 me13 period13 are13 required13 before13 an13 alert13 is13 sent13 or13 a13 specific13 number13 of13 instances13 triggers13 the13 alert13

Page of 36 71 Page of 36 71

Edit13 the13 plugin13 parameters13 to13 meet13 system13 requirements13

13 13 13 13 13 if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 10

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle13

after_count =gt 20

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 30

period =gt 1200

key =gt servicename_alarm_warning

Page of 37 71 Page of 37 71

add_tag =gt servicename_throttled

13 13 13 13 13 13 13 13 13

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 40

period =gt 1500

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

Enrich13 alerts13 with13 message13 detail13 Some13 of13 the13 third-shy‐party13 integraons13 benefit13 from13 having13 components13 of13 the13 raw13 log13 record13 sent13 in13 addion13 to13 the13 raw13 record13 or13 fields13 in13 the13 record13 edited13 transformed13 or13 translated13

Grok13 Logstash13 allows13 the13 log13 record13 to13 be13 broken13 up13 into13 logical13 secons13 and13 each13 secon13 stored13 in13 a13 new13 parameter13 using13 the13 grok13 parameter13 in13 the13 filter13 secon13 Here13 is13 an13 example13 grok13

13 grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

To13 build13 the13 grok13 statement13 Logstash13 offers13 an13 online13 tool13 at13 hcpsgrokdebugherokuappcom13

Translate13 If13 specific13 fields13 in13 the13 log13 record13 are13 coded13 (for13 example13 the13 Bluemix13 app13 name13 is13 oaen13 not13 sent13 in13 the13 syslog13 messages13 instead13 the13 GUID13 is13 sent13 which13 is13 not13 useful)13 then13 the13 filter13 secon13 translate13 plugin13 can13 be13 used13 to13 add13 new13 fields13 by13 holding13 the13 decoded13 versions13 of13 those13 values13

13 translate

field =gt data1

destination =gt data1

override =gt true

Page of 38 71 Page of 38 71

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

dc50f6e4-c052-436a-bf27-95ba392c4228 monverifyservicemanagement

0c677e82-5bfc-4e38-9155-e416249e3ee6 ms-catalog-postpyretic-pomiculture

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

However13 the13 translate13 plugin13 is13 not13 included13 with13 the13 Logstash13 installaon13 In13 the13 next13 secon13 I13 will13 13 describe13 how13 to13 add13 new13 plugins13 to13 the13 Logstash13 instance13

Add13 custom13 plugins13 to13 Logstash13 As13 an13 open13 source13 tool13 Logstash13 encourages13 users13 to13 develop13 content13 for13 all13 users13 to13 share13 Some13 of13 the13 plugins13 that13 were13 created13 by13 our13 community13 include13 a13 translaon13 plugin13 an13 output13 Slack13 plugin13 and13 an13 output13 syslog13 plugin13 Letrsquos13 look13 at13 how13 to13 add13 these13 plugins13 to13 your13 environment13

Filter13 translate13 plugin13 To13 add13 the13 translate13 plugin13

1 Download13 the13 plugin13

2 Copy13 the13 plugin13 file13 logstash-shy‐filter-shy‐translate-shy‐214gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-filter-translate 214

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Output13 Slack13 plugin13 To13 add13 the13 Slack13 plugin13

Page of 39 71 Page of 39 71

1 Download13 the13 plugin13 13

2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐slack-shy‐014gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-output-slack 014

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Output13 syslog13 plugin13 To13 add13 the13 syslog13 plugin13

1 Download13 the13 plugin13 13

2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐syslog-shy‐214gem13 to13 the13 ELK13 Virtual13 Server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-output-syslog 214

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Integrate13 Logstash13 with13 Slack13 To13 send13 generated13 alerts13 to13 Slack13 complete13 these13 steps13

1 Create13 a13 Slack13 integraon13 13

2 Use13 the13 Slack13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 Slack13 integraon13

113 Login13 to13 Slack13 and13 go13 to13 the13 Apps13 page13 13

Page of 40 71 Page of 40 71

13

313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13

13

A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13

13

Page of 42 71 Page of 42 71

The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13

13

The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13

13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

slack

url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5

channel =gt cloud-operations-mon

username =gt nick_cawood

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13

Page of 43 71 Page of 43 71

Page of 44 71 Page of 44 71

In13 Slack13 the13 alerts13 look13 like13 this13

13

Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13

1 Create13 a13 PagerDuty13 integraon13 13

2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13

1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13

13

2 Click13 Add13 New13 Service13

13

313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13

Page of 45 71 Page of 45 71

13

413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13

513 Click13 Add13 Service13

13

The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13

Page of 46 71 Page of 46 71

13

The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt be7eb1ea9c224018923e818d3e891d0b

incident_key =gt logstashhosttype

Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13

Page of 47 71 Page of 47 71

13

Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13

1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13

2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13

3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13

4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13

13 13 http

url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo

http_method =gt post

format =gt json

For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13

13 13 syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

Page of 48 71 Page of 48 71

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13

13

The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13

if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2

This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13

Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13

Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13

Logstash13 severity Shown13 in13 Omnibus13 as

alert Indeterminate

crical Warning

debug 7

emergency Clear

error Minor

informaonal 6

noce Crical

warning Major

Page of 49 71 Page of 49 71

13

Page of 50 71 Page of 50 71

Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13

input

tcp

port =gt 5000

type =gt syslog

filter

mutate

add_field =gt format =gt cf

grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

translate

field =gt data1

destination =gt data1

override =gt true

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

=== Check Alarm Criteria ===

Standard Key Words

if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

=== Alarm Exclusions ===

if [message] =~ (i)sometexthere

mutate remove_tag =gt servicename_alarm

Throttle alarms so we do not hit PD rate limits

Page of 52 71 Page of 52 71

if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 5

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle

after_count =gt 5

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 5

period =gt 1200

key =gt servicename_alarm_warning

add_tag =gt servicename_throttled

Page of 53 71 Page of 53 71

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

output

elasticsearch

hosts =gt localhost

stdout codec =gt rubydebug

if servicename_alarm_err in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

Page of 54 71 Page of 54 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 55 71 Page of 55 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

Page of 56 71 Page of 56 71

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert err message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_error in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

Page of 57 71 Page of 57 71

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 58 71 Page of 58 71

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 59 71 Page of 59 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert error message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

Page of 60 71 Page of 60 71

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

Page of 61 71 Page of 61 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 62 71 Page of 62 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt x

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert fatal message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

Page of 63 71 Page of 63 71

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

Page of 64 71 Page of 64 71

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 65 71 Page of 65 71

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt heavy_exclamation_mark

Page of 66 71 Page of 66 71

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert WARNING message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 67 71 Page of 67 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

Page of 68 71 Page of 68 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 69 71 Page of 69 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

Page of 70 71 Page of 70 71

Page of 71 71 Page of 71 71

Page 8: How to integrate the Elasticsearch Logstash Kibana (ELK ... · How to integrate the Elasticsearch Logstash Kibana (ELK) log analytics stack into IBM Bluemix Nick Cawood IBM Cloud

Create13 an13 SSH13 key13 To13 access13 your13 virtual13 server13 once13 it13 is13 provisioned13 allocate13 an13 SSH13 key13 to13 the13 virtual13 server13

To13 generate13 a13 publicprivate13 RSA13 key13 pair13 follow13 these13 steps13

1 In13 a13 UNIX13 or13 Linuxreg13 window13 run13 the13 following13 command13 specifying13 a13 ltuniquekeynamegt13 of13 your13 your13 choice13 and13 a13 password13 or13 passphrase13 (memorize13 the13 password)$13 ssh-shy‐keygen13 -shy‐t13 rsa13 -shy‐f13 ltuniquekeynamegt13

2 Enter13 the13 passphrase13 (empty13 for13 no13 passphrase)13

3 Enter13 the13 same13 passphrase13 again13

Your13 idenficaon13 has13 been13 saved13 in13 ltuniquekeynamegt13

Your13 public13 key13 has13 been13 saved13 in13 ltuniquekeynamegtpub13

The13 key13 fingerprint13 is13

d6268464506525930ce4632c693c203c13 ltusergtltservergt13

The13 keys13 randomart13 image13 is13

+--[ RSA 2048]----+

|o ++=o |

| E o ooo |

| |

| + o |

| S o |

| o |

| |

| |

| |

+-----------------+

Two13 keys13 are13 created13 public13 and13 private13 The13 public13 key13 contents13 are13 used13 to13 create13 a13 virtual13 server13 with13 a13 public13 IP13 address13 The13 private13 key13 is13 used13 to13 access13 the13 virtual13 server13 from13 a13 client13 machine13

Page of 8 71 Page of 8 71

Provisioning13 a13 virtual13 server13 1 A13 Bluemix13 account13 is13 required13 for13 this13 process13 Sign13 up13 here13 and13 log13 in13 Make13 sure13 your13

environment13 is13 correct13 For13 example13 if13 you13 are13 in13 the13 United13 Kingdom13 select13 the13 United13 Kingdom13 region13 in13 the13 Bluemix13 console13

13

13

2 Add13 a13 virtual13 server13 to13 your13 Bluemix13 environment13 Click13 Run13 Virtual13 Servers13

13

3 Select13 an13 OS13 image13 from13 the13 EXISTING13 list13 For13 ELK13 select13 Centos13 OS13 (use13 latest13 available13 version)13

Page of 9 71 Page of 9 71

4 Name13 the13 virtual13 server13

5 Make13 sure13 Assign13 public13 IP13 address13 is13 checked13

13

6 Click13 Add13 Key13 to13 add13 the13 SSH13 key13 you13 created13 previously13 Copy13 the13 contents13 of13 the13 public13 key13 into13 the13 Public13 Key13 to13 import13 field13

Page of 10 71 Page of 10 71

13

7 Click13 OK13 Then13 click13 CREATE13

The13 virtual13 server13 is13 provisioned13 and13 added13 to13 the13 Bluemix13 dashboard13 under13 VIRTUAL13 SERVERS13

Page of 11 71 Page of 11 71

13

Set13 up13 the13 virtual13 server13

Log13 on13 to13 the13 virtual13 server13 Now13 you13 can13 log13 on13 to13 the13 virtual13 server13 and13 add13 new13 users13 and13 programs13 to13 make13 it13 easier13 to13 access13 the13 virtual13 server13

To13 access13 the13 server13 go13 to13 the13 UNIX13 or13 Linux13 window13 that13 you13 used13 to13 create13 the13 SSH13 key13 In13 the13 directory13 where13 the13 ltuniquekeynamegt13 files13 exist13 run13 the13 following13 command13 referencing13 the13 SSH13 keyname13 the13 public13 IP13 address13 Bluemix13 allocated13 for13 the13 virtual13 server13 and13 the13 password13 for13 the13 SSH13 key13

[userserver ~]$ ssh -i ltuniquekeynamegt ibmcloudltpublicIPaddressgt

Enter13 the13 passphrase13 for13 the13 key13 ltuniquekeynamegt13

Last login Tue Jul 5 125546 2016 from 108-210-96-231lightspeedrlghncsbcglobalnet

[ibmcloudelkforcase ~]$

ibmcloud13 is13 the13 default13 user13 for13 the13 Bluemix13 virtual13 servers13 and13 allows13 SSH13 to13 access13 the13 server13

Acces13 the13 root13 user13 To13 access13 the13 root13 user13 log13 on13 as13 ibmcloud13 user13 and13 run13 the13 following13 command13

[ibmcloudelkforcase ~]$ sudo bash

[rootelkforcase ibmcloud]

The13 root13 user13 can13 be13 used13 to13 add13 programs13 to13 the13 server13 and13 create13 and13 edit13 users13 as13 required13 to13 access13 applicaons13 installed13 on13 the13 virtual13 server13

Page of 12 71 Page of 12 71

Copy13 files13 to13 the13 virtual13 server13 To13 move13 installaon13 files13 to13 the13 server13 go13 to13 the13 UNIX13 or13 Linux13 window13 you13 used13 to13 create13 the13 SSH13 key13 and13 the13 directory13 where13 the13 ltuniquekeynamegt13 files13 exist13 Move13 the13 installaon13 files13 to13 the13 same13 locaon13

Run13 the13 following13 command13 referencing13 the13 SSH13 keyname13 the13 public13 IP13 address13 Bluemix13 allocated13 for13 the13 virtual13 server13 and13 the13 password13 for13 the13 SSH13 key13

[userserver ~]$ scp ndashi ltuniquekeynamegt ltINSTALLATIONFILENAMEgt ibmcloudltpublicIPaddressgthomeibmcloud

Enter13 the13 passphrase13 for13 key13 cloudkey313

This13 copies13 the13 file13 using13 the13 ibmcloud13 user13 to13 the13 homeibmcloud13 directory13 on13 the13 virtual13 server13

Install13 and13 access13 the13 virtual13 server13 via13 VNC13 server13 Using13 the13 ssh13 command13 to13 access13 the13 virtual13 server13 is13 not13 the13 most13 stable13 method13 and13 does13 not13 offer13 X-shy‐Windows13 support13 Therefore13 we13 suggest13 that13 you13 use13 a13 VNC13 client13 to13 access13 the13 virtual13 server13

Installing13 a13 VNC13 client13 should13 be13 done13 as13 the13 root13 user13 Use13 the13 yum13 command13 as13 follows13

[ibmcloudelkforcase ~]$ yum groupinstall GNOME Desktop

[ibmcloudelkforcase ~]$ yum install tigervnc-server

These13 commands13 first13 install13 a13 desktop13 style13 OS13 user13 interface13 and13 then13 a13 VNC13 client13 to13 remotely13 access13 the13 desktop13 style13 OS13

Then13 as13 either13 the13 root13 user13 or13 another13 user13 (example13 below13 uses13 elk)13 run13 the13 following13 command13 to13 create13 the13 VNC13 client13 connecon13

[elkelkforcase ~]$ vncserver

New elkforcase2 (elk) desktop is elkforcase2

Starting applications specified in homeelkvncxstartup

Log file is homeelkvncelkforcase2log

[elkelkforcase ~]$

Install13 a13 VNC13 client13 on13 the13 client13 machine13 used13 to13 access13 the13 Internet13 and13 (in13 the13 example13 above)13 use13 the13 command13 ltpublicIPadddressgt213 and13 the13 password13 for13 ELK13 to13 access13 the13 virtual13 server13

Using13 a13 VNC13 client13 means13 that13 you13 can13 leave13 long13 duraon13 installaons13 running13 without13 fear13 of13 the13 terminal13 losing13 connecon13 and13 the13 installaons13 failing13

Install13 the13 ELK13 stack13 Download13 the13 source13 soaware13 code13 for13 the13 ELK13 stack13 from13 the13 ELK13 website13 13

At13 the13 me13 of13 wring13 the13 downloaded13 soaware13 files13 were13

Page of 13 71 Page of 13 71

13

Transfer13 the13 files13 to13 the13 virtual13 server13 (using13 the13 scp13 method13 detailed13 above)13

Install13 Elas0csearch13 As13 a13 root13 user13 create13 a13 non-shy‐root13 user13 with13 the13 adduser13 command13

Create13 a13 directory13 to13 install13 Elascsearch13 into13 and13 untar13 the13 contents13 of13 the13 Elascsearch13 install13 file13 into13 that13 directory13

[elkelkforcase]$ cd home

[elkelkforcase]$ mkdir elasticsearch

[elkelkforcase]$ cp homeibmcloudelasticsearch-233targz homeelasticsearch

[elkelkforcase]$ cd elasticsearch

[elkelkforcase]$ gunzip elasticsearch-233targz

[elkelkforcase]$ untar elasticsearch-233tar

Elascsearch13 is13 now13 installed13

Start13 Elas0csearch13 To13 start13 Elascsearch13 go13 to13 the13 bin13 directory13 as13 the13 non-shy‐root13 user13 when13 you13 installed13 it13 and13 run13 the13 elascsearch13 executable13

[elkelkforcase]$ cd homeelasticsearchelasticsearch-233bin

[elkelkforcase]$ elasticsearch amp

[2] 22211

[elkelkforcase]$ [2016-07-05 133857730][INFO ][node] [Arsenal] version[233] pid[22211] build[218bdf12016-05-17T154004Z]

[2016-07-05 133857740][INFO ][node] [Arsenal] initializing

[2016-07-05 133858984][INFO ][plugins] [Arsenal] modules [reindex lang-expression lang-groovy] plugins [] sites []

[2016-07-05 133859146][INFO ][env] [Arsenal] using [1] data paths mounts [[ (rootfs)]] net usable_space [155gb] net total_space [199gb] spins [unknown] types [rootfs]

[2016-07-05 133859146][INFO ][env] [Arsenal] heap size [10156mb] compressed ordinary object pointers [true]

Page of 14 71 Page of 14 71

[2016-07-05 133859147][WARN ][env] [Arsenal] max file descriptors [4096] for elasticsearch process likely too low consider increasing to at least [65536]

[2016-07-05 133903526][INFO ][node] [Arsenal] initialized

[2016-07-05 133903526][INFO ][node] [Arsenal] starting

To13 test13 that13 Elascsearch13 is13 running13 correctly13 open13 the13 browser13 in13 the13 VNC13 client13 and13 run13 localhost920013

13

Install13 Kibana13 ELK13 recommends13 installing13 and13 running13 Kibana13 as13 root13 13

1 As13 root13 user13 create13 an13 installaon13 directory13 and13 untar13 the13 installaon13 file13

[rootelkforcase]$ cd home

[rootelkforcase]$ mkdir kibana

[rootelkforcase]$ cp homeibmcloudkibana-451-linux-x64targz homekibana

[rootelkforcase]$ cd kibana

[rootelkforcase]$ gunzip kibana-451-linux-x64targz

[rootelkforcase]$ untar kibana-451-linux-x64tar

2 Modify13 the13 manifestyml13 configuraon13 file13 Go13 to13 the13 kibana-shy‐451-shy‐linux-shy‐x6413 directory13

[rootelkforcase]$ cd homekibanakibana-451-linux-x64

3 Edit13 the13 manifestyml13 file13 so13 the13 contents13 are13

applicaons13

-shy‐ name13 kibana-shy‐in-shy‐bluemix13

memory13 128M13

host13 kibana-shy‐in-shy‐bluemix13

buildpack13 hcpsgithubcomcloudfoundry-shy‐communitynginx-shy‐buildpackgit13

Page of 15 71 Page of 15 71

4 Edit13 the13 configkibanayml13 file13 so13 the13 contents13 are13

Kibana is served by a back end server This controls which port to use

port 5601

The host to bind the server to

host 0000

If you are running kibana behind a proxy and want to mount it at a path

specify that path here The basePath cant end in a slash

serverbasePath

The maximum payload size in bytes on incoming server requests

servermaxPayloadBytes 1048576

The Elasticsearch instance to use for all your queries

elasticsearchurl httplocalhost9200

preserve_elasticsearch_host true will send the hostname specified in `elasticsearch` If you set it to false

then the host you use to connect to this Kibana instance will be sent

elasticsearchpreserveHost true

Kibana uses an index in Elasticsearch to store saved searches visualizations

and dashboards It will create a new index if it doesnt already exist

kibanaindex kibana

The default application to load

kibanadefaultAppId discover

If your Elasticsearch is protected with basic auth these are the user credentials

used by the Kibana server to perform maintenance on the kibana_index at startup Your

Kibana users will still need to authenticate with Elasticsearch (which is proxied through

the Kibana server)

Page of 16 71 Page of 16 71

elasticsearchusername user

elasticsearchpassword pass

SSL for outgoing requests from the Kibana Server to the browser (PEM formatted)

serversslcert pathtoyourservercrt

serversslkey pathtoyourserverkey

Optional setting to validate that your Elasticsearch backend uses the same key files (PEM formatted)

elasticsearchsslcert pathtoyourclientcrt

elasticsearchsslkey pathtoyourclientkey

If you need to provide a CA certificate for your Elasticsearch instance put

the path of the pem file here

elasticsearchsslca pathtoyourCApem

Set to false to have a complete disregard for the validity of the SSL

certificate

elasticsearchsslverify true

Time in milliseconds to wait for elasticsearch to respond to pings defaults to

request_timeout setting

elasticsearchpingTimeout 1500

Time in milliseconds to wait for responses from the back end or elasticsearch

This must be gt 0

elasticsearchrequestTimeout 30000

Time in milliseconds for Elasticsearch to wait for responses from shards

Set to 0 to disable

elasticsearchshardTimeout 0

Time in milliseconds to wait for Elasticsearch at Kibana startup before retrying

elasticsearchstartupTimeout 5000

elasticsearchstartupTimeout 30000

Page of 17 71 Page of 17 71

Set the path to where you would like the process id file to be created

pidfile varrunkibanapid

If you would like to send the log output to a file you can set the path below

loggingdest stdout

Set this to true to suppress all logging output

loggingsilent false

Set this to true to suppress all logging output except for error messages

loggingquiet false

Set this to true to log all events including system usage information and all requests

loggingverbose false

Kibana13 is13 now13 installed13

Start13 Kibana13 To13 start13 Kibana13 go13 to13 the13 bin13 directory13 and13 run13 the13 kibana13 executable13

[rootelkforcase]$

[rootelkforcase]$ cd homekibanakibana-451-linux-x64bin

[rootelkforcase]$ kibana amp

[1] 22283

[rootelkforcase] log [095900542] [info][status][pluginkibana] Status changed from uninitialized to green - Ready

log [095900600] [info][status][pluginelasticsearch] Status changed from uninitialized to yellow - Waiting for Elasticsearch

log [095900689] [info][status][pluginkbn_vislib_vis_types] Status changed from uninitialized to green - Ready

log [095900717] [info][status][pluginmarkdown_vis] Status changed from uninitialized to green - Ready

log [095900721] [info][status][pluginmetric_vis] Status changed from uninitialized to green - Ready

log [095900814] [info][status][pluginspyModes] Status changed from uninitialized to green - Ready

log [095900934] [info][status][pluginstatusPage] Status changed from uninitialized to green - Ready

Page of 18 71 Page of 18 71

log [095900941] [info][status][plugintable_vis] Status changed from uninitialized to green - Ready

log [095901067] [info][listening] Server running at http00005601

log [095906083] [info][status][pluginelasticsearch] Status changed from yellow to yellow - No existing Kibana index found

log [095908942] [info][status][pluginelasticsearch] Status changed from yellow to green - Kibana index ready

Kibana13 is13 now13 started13

Broadcast13 Bluemix13 app13 log13 records13 Bluemix13 apps13 (which13 are13 shown13 in13 the13 US13 South13 Bluemix13 region13 of13 the13 System13 Architecture13 diagram)13 will13 produce13 logs13 in13 a13 syslog13 rfc542413 format13 using13 a13 Bluemix13 tool13 called13 Loggregator13

For13 Logstash13 to13 receive13 these13 logs13 each13 Bluemix13 app13 needs13 to13 broadcast13 these13 logs13 To13 do13 that13 run13 the13 following13 commands13 on13 each13 Bluemix13 app13 using13 the13 public13 IP13 address13 of13 the13 ELK13 virtual13 server13 the13 ELK13 tag13 to13 trace13 the13 process13 and13 insert13 the13 name13 of13 each13 app13 in13 turn13

cf cups elk -l syslogltpublicIPaddress5000 cf bind-service ltmyappgt elk

cf restage ltmyappgt

Install13 Logstash13 To13 install13 Logstash13 as13 root13 user13 create13 a13 directory13 for13 the13 installaon13 and13 untar13 the13 installaon13 file13 there13

13 13 [rootelkforcase]$ cd home

[rootelkforcase]$ mkdir logstash

[rootelkforcase]$ cp homeibmcloudlogstash-233targz homelogstash

[rootelkforcase]$ cd logstash

[rootelkforcase]$ gunzip logstash-233targz

[rootelkforcase]$ untar logstash-233tar

Write13 Logstash13 configura0on13 Logstash13 requires13 a13 configuraon13 file13 to13 be13 wricen13 to13 receive13 transform13 and13 output13 the13 log13 records13 to13 Elascsearch13 This13 secon13 explains13 the13 basic13 requirements13 of13 processing13 Bluemix13 app13 log13 records13 into13 Elascsearch13 (more13 complex13 configuraons13 are13 detailed13 later13 in13 this13 document13 for13 the13 integraons13 with13 third-shy‐party13 tools)13

Create13 a13 conf13 file13 as13 root13 user13

[rootelkforcase]$ cd homelogstashlogstash-233

[rootelkforcase]$ mkdir config

Page of 19 71 Page of 19 71

[rootelkforcase]$ cd config

[rootelkforcase]$ touch loggregatorconf

Edit13 the13 loggregator13 file13 with13 the13 following13 contents13

13 Input13 secon13 defines13 which13 log13 records13 to13 receive13

13 port13 and13 type13 reflect13 method13 selected13 for13 Cloud13 Foundry13 to13 broadcast13 the13 records13

input

tcp

port =gt 5000

type =gt syslog

13 Filter13 secon13 ndash13 transforms13 the13 data13

filter

Add a field to identify the records as Cloud Foundry based

mutate

add_field =gt format =gt cf

13 Output13 secon13 ndash13 sends13 the13 data13 to13 its13 desnaon(s)13

output

13 Use13 the13 elascsearch13 plugin13 to13 send13 the13 data13 to13 the13 elascsearch13 instance13

13 elasticsearch

hosts =gt localhost

stdout codec =gt rubydebug

13 End13 of13 Config13

Save13 the13 file13 Verify13 the13 format13 of13 the13 file13 is13 acceptable13 by13 running13 the13 following13 commands13

[rootelkforcase]$ cd homelogstashlogstash-233

[rootelkforcase]$ binlogstash agent --configtest -f configloggregatorconf

Page of 20 71 Page of 20 71

Configuraon13 is13 OK13

Start13 Logstash13 Aaer13 you13 have13 wricen13 and13 verified13 the13 Logstash13 configuraon13 you13 can13 start13 Logstash13 to13 process13 records13 from13 the13 Bluemix13 apps13 (via13 Cloud13 Foundry)13 Run13 the13 following13 as13 root13 user13

[rootelkforcase]$ cd homelogstashlogstash-233

[rootelkforcase]$ binlogstash agent -f configloggregatorconf

Logstash13 is13 now13 started13

Monitor13 Logstash13 processing13 Once13 Logstash13 is13 running13 and13 log13 records13 are13 being13 produced13 by13 the13 Bluemix13 apps13 you13 can13 monitor13 progress13 in13 the13 same13 window13 that13 was13 used13 to13 start13 Logstash13 Output13 is13 similar13 to13 the13 following13

13

The13 log13 records13 being13 broadcast13 from13 the13 Bluemix13 apps13 are13 now13 sent13 to13 Elascsearch13 from13 the13 Logstash13 output13 configuraon13 Records13 are13 sent13 in13 real13 me13 If13 Logstash13 is13 not13 running13 while13 log13 records13 are13 being13 produced13 then13 these13 records13 will13 not13 be13 processed13 by13 Logstash13 and13 will13 not13 be13 stored13 in13 Elascsearch13

Page of 21 71 Page of 21 71

Use13 ELK13 The13 log13 records13 from13 the13 Bluemix13 apps13 are13 now13 being13 stored13 and13 indexed13 by13 the13 Elascsearch13 component13 This13 data13 can13 be13 accessed13 via13 the13 Kibana13 applicaon13

Kibana13 Discover13 page13 Using13 a13 web13 browser13 either13 directly13 from13 a13 client13 or13 using13 the13 VNC13 client13 access13 Kibana13 using13 the13 publicIPaddress13 of13 the13 ELK13 virtual13 server13

hcpltpublicIPaddressgt560113

No13 userpassword13 credenals13 are13 required13

Kibana13 defaults13 to13 a13 search13 of13 all13 records13 ()13 in13 the13 last13 1513 minutes13 and13 returns13 the13 results13

13

More13 complex13 searches13 are13 possible13 by13 changing13 the13 me13 filter13 and13 supplying13 search13 queries13

Change13 the13 0me13 filter13 To13 change13 the13 me13 ranges13 click13 the13 clock13 symbol13 in13 the13 top-shy‐right13 of13 the13 Kibana13 Discover13 page13

13

You13 will13 see13 the13 following13 opons13

13

Page of 22 71 Page of 22 71

A13 variety13 of13 quick13 me13 ranges13 can13 be13 selected13 or13 you13 can13 switch13 to13 relave13 or13 absolute13 me13 ranges13

Quick13 me13 ranges13 allow13 switching13 between13 last13 X13 minutes13 to13 last13 Y13 hours13 or13 Z13 days13 etc13

Relave13 me13 ranges13 allow13 more13 specific13 versions13 of13 the13 Quick13 opons13 from13 specific13 minuteshoursdays13 to13 now13

13

Absolute13 me13 ranges13 specify13 the13 start13 and13 end13 mes13 for13 the13 searches13

13

Custom13 search13 queries13 You13 can13 use13 the13 text13 window13 of13 the13 Discover13 page13 to13 search13 for13 matches13 within13 the13 log13 records13 You13 can13 use13 Apache13 Lucene13 or13 Elascsearch13 query13 DSL13 syntax13

13

Page of 23 71 Page of 23 71

Saved13 searches13 You13 can13 save13 any13 custom13 search13 syntax13 and13 choice13 of13 me13 range13 for13 future13 ease13 of13 use13 Any13 relave13 or13 absolute13 me13 frame13 is13 saved13 as13 part13 of13 the13 search13 13

To13 save13 a13 search13

1 Set13 up13 and13 run13 the13 required13 syntax13 and13 me13 frame13

13

2 Click13 the13 Save13 Search13 icon13

13

3 Type13 in13 a13 name13 for13 the13 saved13 search13

Page of 24 71 Page of 24 71

13

7 To13 delete13 or13 update13 saved13 searches13 click13 13 manage13 saved13 searches13

13

Use13 the13 edit13 icons13 to13 make13 any13 changes13

Page of 26 71 Page of 26 71

13

Build13 simple13 Kibana13 histograms13 Kibana13 allows13 you13 to13 build13 charts13 and13 dashboards13 based13 on13 the13 search13 capability13 offered13 by13 the13 Discover13 page13 Saved13 searches13 can13 be13 used13 as13 the13 basis13 for13 charts13 and13 dashboards13 built13 using13 those13 charts13 To13 build13 a13 simple13 histogram13

1 Select13 the13 Visualize13 page13

13

2 Select13 Ver0cal13 Bar13 Chart13

13

3 To13 use13 a13 saved13 search13 click13 From13 a13 saved13 search13

Page of 27 71 Page of 27 71

13

4 Select13 a13 previously13 saved13 search13

13

5 Select13 to13 edit13 the13 x-shy‐axis

6 Edit13 the13 fields13 with13 informaon13 about13 the13 aggregaon13 field13 interval13 and13 custom13 label13

Page of 28 71 Page of 28 71

13

Page of 29 71 Page of 29 71

7 Click13 the13 green13 arrow13 The13 histogram13 is13 created13 and13 run13 based13 on13 the13 saved13 search

13

8 To13 save13 the13 histogram13 click13 the13 Save13 Visualiza0on13 icon13 and13 add13 a13 tle13

13

13

9 Click13 Save13

10 To13 view13 the13 visualisaon13 click13 the13 Dashboard13 page13

Page of 30 71 Page of 30 71

13

11 Click13 the13 Add13 Visualiza0on13 icon13

13

12 Select13 the13 visiualizaon13 that13 you13 just13 created13

13

Page of 31 71 Page of 31 71

13

13 To13 save13 the13 dashboard13 click13 the13 Save13 Dashboard13 icon13

Page of 32 71 Page of 32 71

13

14 Add13 a13 tle13 and13 select13 whether13 to13 store13 the13 selected13 me13 frame13 with13 the13 dashboard13

13

15 Click13 Save13

16 You13 can13 edit13 the13 shape13 and13 number13 of13 charts13 and13 save13 the13 dashboard13

17 To13 re-shy‐run13 the13 dashboard13 select13 the13 Dashboard13 page13 and13 click13 the13 Load13 Saved13 Dashboard13 icon13

Page of 33 71 Page of 33 71

13

18 Select13 the13 saved13 dashboard13 from13 the13 list13

13

19 The13 dashboard13 runs13

13

Page of 34 71 Page of 34 71

Page of 35 71 Page of 35 71

Aler0ng13 with13 ELK13 ELK13 examines13 the13 log13 records13 in13 process13 to13 send13 alerts13 to13 third-shy‐party13 alerng13 event13 management13 and13 social13 media13 applicaons13 As13 shown13 previously13 in13 the13 ELK13 architecture13 the13 ELK13 strategy13 for13 alerng13 is13 to13 send13 alerts13 from13 the13 Logstash13 component13 directly13 Therefore13 log13 records13 and13 alerts13 are13 sent13 to13 both13 Elascsearch13 and13 third-shy‐party13 tools13 at13 the13 same13 instant13 using13 the13 output13 secon13 of13 the13 Logstash13 configuraon13 Any13 alerts13 are13 generated13 during13 the13 filter13 secon13 processing13 of13 the13 Logstash13 configuraon13

Create13 alerts13 in13 Logstash13 The13 Logstash13 filter13 secon13 is13 used13 to13 trigger13 alerts13 based13 on13 the13 log13 records13 being13 processed13 For13 example13 log13 records13 can13 be13 checked13 for13 errors13 warnings13 crashed13 or13 stopped13 messages13 and13 generate13 an13 alert13

Here13 is13 an13 example13 Logstash13 filter13 code13

13 if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

The13 specific13 tags13 are13 set13 against13 the13 log13 records13 to13 trigger13 the13 alert13 later13 in13 the13 configuraon13 in13 the13 output13 secon13

Control13 the13 number13 of13 generated13 alerts13 using13 the13 Logstash13 throcle13 plugin13 so13 that13 either13 a13 lot13 of13 instances13 in13 a13 fixed13 me13 period13 are13 required13 before13 an13 alert13 is13 sent13 or13 a13 specific13 number13 of13 instances13 triggers13 the13 alert13

Page of 36 71 Page of 36 71

Edit13 the13 plugin13 parameters13 to13 meet13 system13 requirements13

13 13 13 13 13 if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 10

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle13

after_count =gt 20

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 30

period =gt 1200

key =gt servicename_alarm_warning

Page of 37 71 Page of 37 71

add_tag =gt servicename_throttled

13 13 13 13 13 13 13 13 13

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 40

period =gt 1500

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

Enrich13 alerts13 with13 message13 detail13 Some13 of13 the13 third-shy‐party13 integraons13 benefit13 from13 having13 components13 of13 the13 raw13 log13 record13 sent13 in13 addion13 to13 the13 raw13 record13 or13 fields13 in13 the13 record13 edited13 transformed13 or13 translated13

Grok13 Logstash13 allows13 the13 log13 record13 to13 be13 broken13 up13 into13 logical13 secons13 and13 each13 secon13 stored13 in13 a13 new13 parameter13 using13 the13 grok13 parameter13 in13 the13 filter13 secon13 Here13 is13 an13 example13 grok13

13 grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

To13 build13 the13 grok13 statement13 Logstash13 offers13 an13 online13 tool13 at13 hcpsgrokdebugherokuappcom13

Translate13 If13 specific13 fields13 in13 the13 log13 record13 are13 coded13 (for13 example13 the13 Bluemix13 app13 name13 is13 oaen13 not13 sent13 in13 the13 syslog13 messages13 instead13 the13 GUID13 is13 sent13 which13 is13 not13 useful)13 then13 the13 filter13 secon13 translate13 plugin13 can13 be13 used13 to13 add13 new13 fields13 by13 holding13 the13 decoded13 versions13 of13 those13 values13

13 translate

field =gt data1

destination =gt data1

override =gt true

Page of 38 71 Page of 38 71

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

dc50f6e4-c052-436a-bf27-95ba392c4228 monverifyservicemanagement

0c677e82-5bfc-4e38-9155-e416249e3ee6 ms-catalog-postpyretic-pomiculture

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

However13 the13 translate13 plugin13 is13 not13 included13 with13 the13 Logstash13 installaon13 In13 the13 next13 secon13 I13 will13 13 describe13 how13 to13 add13 new13 plugins13 to13 the13 Logstash13 instance13

Add13 custom13 plugins13 to13 Logstash13 As13 an13 open13 source13 tool13 Logstash13 encourages13 users13 to13 develop13 content13 for13 all13 users13 to13 share13 Some13 of13 the13 plugins13 that13 were13 created13 by13 our13 community13 include13 a13 translaon13 plugin13 an13 output13 Slack13 plugin13 and13 an13 output13 syslog13 plugin13 Letrsquos13 look13 at13 how13 to13 add13 these13 plugins13 to13 your13 environment13

Filter13 translate13 plugin13 To13 add13 the13 translate13 plugin13

1 Download13 the13 plugin13

2 Copy13 the13 plugin13 file13 logstash-shy‐filter-shy‐translate-shy‐214gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-filter-translate 214

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Output13 Slack13 plugin13 To13 add13 the13 Slack13 plugin13

Page of 39 71 Page of 39 71

1 Download13 the13 plugin13 13

2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐slack-shy‐014gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-output-slack 014

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Output13 syslog13 plugin13 To13 add13 the13 syslog13 plugin13

1 Download13 the13 plugin13 13

2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐syslog-shy‐214gem13 to13 the13 ELK13 Virtual13 Server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-output-syslog 214

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Integrate13 Logstash13 with13 Slack13 To13 send13 generated13 alerts13 to13 Slack13 complete13 these13 steps13

1 Create13 a13 Slack13 integraon13 13

2 Use13 the13 Slack13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 Slack13 integraon13

113 Login13 to13 Slack13 and13 go13 to13 the13 Apps13 page13 13

Page of 40 71 Page of 40 71

13

313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13

13

A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13

13

Page of 42 71 Page of 42 71

The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13

13

The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13

13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

slack

url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5

channel =gt cloud-operations-mon

username =gt nick_cawood

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13

Page of 43 71 Page of 43 71

Page of 44 71 Page of 44 71

In13 Slack13 the13 alerts13 look13 like13 this13

13

Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13

1 Create13 a13 PagerDuty13 integraon13 13

2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13

1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13

13

2 Click13 Add13 New13 Service13

13

313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13

Page of 45 71 Page of 45 71

13

413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13

513 Click13 Add13 Service13

13

The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13

Page of 46 71 Page of 46 71

13

The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt be7eb1ea9c224018923e818d3e891d0b

incident_key =gt logstashhosttype

Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13

Page of 47 71 Page of 47 71

13

Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13

1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13

2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13

3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13

4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13

13 13 http

url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo

http_method =gt post

format =gt json

For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13

13 13 syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

Page of 48 71 Page of 48 71

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13

13

The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13

if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2

This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13

Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13

Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13

Logstash13 severity Shown13 in13 Omnibus13 as

alert Indeterminate

crical Warning

debug 7

emergency Clear

error Minor

informaonal 6

noce Crical

warning Major

Page of 49 71 Page of 49 71

13

Page of 50 71 Page of 50 71

Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13

input

tcp

port =gt 5000

type =gt syslog

filter

mutate

add_field =gt format =gt cf

grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

translate

field =gt data1

destination =gt data1

override =gt true

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

=== Check Alarm Criteria ===

Standard Key Words

if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

=== Alarm Exclusions ===

if [message] =~ (i)sometexthere

mutate remove_tag =gt servicename_alarm

Throttle alarms so we do not hit PD rate limits

Page of 52 71 Page of 52 71

if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 5

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle

after_count =gt 5

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 5

period =gt 1200

key =gt servicename_alarm_warning

add_tag =gt servicename_throttled

Page of 53 71 Page of 53 71

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

output

elasticsearch

hosts =gt localhost

stdout codec =gt rubydebug

if servicename_alarm_err in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

Page of 54 71 Page of 54 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 55 71 Page of 55 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

Page of 56 71 Page of 56 71

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert err message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_error in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

Page of 57 71 Page of 57 71

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 58 71 Page of 58 71

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 59 71 Page of 59 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert error message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

Page of 60 71 Page of 60 71

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

Page of 61 71 Page of 61 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 62 71 Page of 62 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt x

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert fatal message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

Page of 63 71 Page of 63 71

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

Page of 64 71 Page of 64 71

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 65 71 Page of 65 71

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt heavy_exclamation_mark

Page of 66 71 Page of 66 71

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert WARNING message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 67 71 Page of 67 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

Page of 68 71 Page of 68 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 69 71 Page of 69 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

Page of 70 71 Page of 70 71

Page of 71 71 Page of 71 71

Page 9: How to integrate the Elasticsearch Logstash Kibana (ELK ... · How to integrate the Elasticsearch Logstash Kibana (ELK) log analytics stack into IBM Bluemix Nick Cawood IBM Cloud

Provisioning13 a13 virtual13 server13 1 A13 Bluemix13 account13 is13 required13 for13 this13 process13 Sign13 up13 here13 and13 log13 in13 Make13 sure13 your13

environment13 is13 correct13 For13 example13 if13 you13 are13 in13 the13 United13 Kingdom13 select13 the13 United13 Kingdom13 region13 in13 the13 Bluemix13 console13

13

13

2 Add13 a13 virtual13 server13 to13 your13 Bluemix13 environment13 Click13 Run13 Virtual13 Servers13

13

3 Select13 an13 OS13 image13 from13 the13 EXISTING13 list13 For13 ELK13 select13 Centos13 OS13 (use13 latest13 available13 version)13

Page of 9 71 Page of 9 71

4 Name13 the13 virtual13 server13

5 Make13 sure13 Assign13 public13 IP13 address13 is13 checked13

13

6 Click13 Add13 Key13 to13 add13 the13 SSH13 key13 you13 created13 previously13 Copy13 the13 contents13 of13 the13 public13 key13 into13 the13 Public13 Key13 to13 import13 field13

Page of 10 71 Page of 10 71

13

7 Click13 OK13 Then13 click13 CREATE13

The13 virtual13 server13 is13 provisioned13 and13 added13 to13 the13 Bluemix13 dashboard13 under13 VIRTUAL13 SERVERS13

Page of 11 71 Page of 11 71

13

Set13 up13 the13 virtual13 server13

Log13 on13 to13 the13 virtual13 server13 Now13 you13 can13 log13 on13 to13 the13 virtual13 server13 and13 add13 new13 users13 and13 programs13 to13 make13 it13 easier13 to13 access13 the13 virtual13 server13

To13 access13 the13 server13 go13 to13 the13 UNIX13 or13 Linux13 window13 that13 you13 used13 to13 create13 the13 SSH13 key13 In13 the13 directory13 where13 the13 ltuniquekeynamegt13 files13 exist13 run13 the13 following13 command13 referencing13 the13 SSH13 keyname13 the13 public13 IP13 address13 Bluemix13 allocated13 for13 the13 virtual13 server13 and13 the13 password13 for13 the13 SSH13 key13

[userserver ~]$ ssh -i ltuniquekeynamegt ibmcloudltpublicIPaddressgt

Enter13 the13 passphrase13 for13 the13 key13 ltuniquekeynamegt13

Last login Tue Jul 5 125546 2016 from 108-210-96-231lightspeedrlghncsbcglobalnet

[ibmcloudelkforcase ~]$

ibmcloud13 is13 the13 default13 user13 for13 the13 Bluemix13 virtual13 servers13 and13 allows13 SSH13 to13 access13 the13 server13

Acces13 the13 root13 user13 To13 access13 the13 root13 user13 log13 on13 as13 ibmcloud13 user13 and13 run13 the13 following13 command13

[ibmcloudelkforcase ~]$ sudo bash

[rootelkforcase ibmcloud]

The13 root13 user13 can13 be13 used13 to13 add13 programs13 to13 the13 server13 and13 create13 and13 edit13 users13 as13 required13 to13 access13 applicaons13 installed13 on13 the13 virtual13 server13

Page of 12 71 Page of 12 71

Copy13 files13 to13 the13 virtual13 server13 To13 move13 installaon13 files13 to13 the13 server13 go13 to13 the13 UNIX13 or13 Linux13 window13 you13 used13 to13 create13 the13 SSH13 key13 and13 the13 directory13 where13 the13 ltuniquekeynamegt13 files13 exist13 Move13 the13 installaon13 files13 to13 the13 same13 locaon13

Run13 the13 following13 command13 referencing13 the13 SSH13 keyname13 the13 public13 IP13 address13 Bluemix13 allocated13 for13 the13 virtual13 server13 and13 the13 password13 for13 the13 SSH13 key13

[userserver ~]$ scp ndashi ltuniquekeynamegt ltINSTALLATIONFILENAMEgt ibmcloudltpublicIPaddressgthomeibmcloud

Enter13 the13 passphrase13 for13 key13 cloudkey313

This13 copies13 the13 file13 using13 the13 ibmcloud13 user13 to13 the13 homeibmcloud13 directory13 on13 the13 virtual13 server13

Install13 and13 access13 the13 virtual13 server13 via13 VNC13 server13 Using13 the13 ssh13 command13 to13 access13 the13 virtual13 server13 is13 not13 the13 most13 stable13 method13 and13 does13 not13 offer13 X-shy‐Windows13 support13 Therefore13 we13 suggest13 that13 you13 use13 a13 VNC13 client13 to13 access13 the13 virtual13 server13

Installing13 a13 VNC13 client13 should13 be13 done13 as13 the13 root13 user13 Use13 the13 yum13 command13 as13 follows13

[ibmcloudelkforcase ~]$ yum groupinstall GNOME Desktop

[ibmcloudelkforcase ~]$ yum install tigervnc-server

These13 commands13 first13 install13 a13 desktop13 style13 OS13 user13 interface13 and13 then13 a13 VNC13 client13 to13 remotely13 access13 the13 desktop13 style13 OS13

Then13 as13 either13 the13 root13 user13 or13 another13 user13 (example13 below13 uses13 elk)13 run13 the13 following13 command13 to13 create13 the13 VNC13 client13 connecon13

[elkelkforcase ~]$ vncserver

New elkforcase2 (elk) desktop is elkforcase2

Starting applications specified in homeelkvncxstartup

Log file is homeelkvncelkforcase2log

[elkelkforcase ~]$

Install13 a13 VNC13 client13 on13 the13 client13 machine13 used13 to13 access13 the13 Internet13 and13 (in13 the13 example13 above)13 use13 the13 command13 ltpublicIPadddressgt213 and13 the13 password13 for13 ELK13 to13 access13 the13 virtual13 server13

Using13 a13 VNC13 client13 means13 that13 you13 can13 leave13 long13 duraon13 installaons13 running13 without13 fear13 of13 the13 terminal13 losing13 connecon13 and13 the13 installaons13 failing13

Install13 the13 ELK13 stack13 Download13 the13 source13 soaware13 code13 for13 the13 ELK13 stack13 from13 the13 ELK13 website13 13

At13 the13 me13 of13 wring13 the13 downloaded13 soaware13 files13 were13

Page of 13 71 Page of 13 71

13

Transfer13 the13 files13 to13 the13 virtual13 server13 (using13 the13 scp13 method13 detailed13 above)13

Install13 Elas0csearch13 As13 a13 root13 user13 create13 a13 non-shy‐root13 user13 with13 the13 adduser13 command13

Create13 a13 directory13 to13 install13 Elascsearch13 into13 and13 untar13 the13 contents13 of13 the13 Elascsearch13 install13 file13 into13 that13 directory13

[elkelkforcase]$ cd home

[elkelkforcase]$ mkdir elasticsearch

[elkelkforcase]$ cp homeibmcloudelasticsearch-233targz homeelasticsearch

[elkelkforcase]$ cd elasticsearch

[elkelkforcase]$ gunzip elasticsearch-233targz

[elkelkforcase]$ untar elasticsearch-233tar

Elascsearch13 is13 now13 installed13

Start13 Elas0csearch13 To13 start13 Elascsearch13 go13 to13 the13 bin13 directory13 as13 the13 non-shy‐root13 user13 when13 you13 installed13 it13 and13 run13 the13 elascsearch13 executable13

[elkelkforcase]$ cd homeelasticsearchelasticsearch-233bin

[elkelkforcase]$ elasticsearch amp

[2] 22211

[elkelkforcase]$ [2016-07-05 133857730][INFO ][node] [Arsenal] version[233] pid[22211] build[218bdf12016-05-17T154004Z]

[2016-07-05 133857740][INFO ][node] [Arsenal] initializing

[2016-07-05 133858984][INFO ][plugins] [Arsenal] modules [reindex lang-expression lang-groovy] plugins [] sites []

[2016-07-05 133859146][INFO ][env] [Arsenal] using [1] data paths mounts [[ (rootfs)]] net usable_space [155gb] net total_space [199gb] spins [unknown] types [rootfs]

[2016-07-05 133859146][INFO ][env] [Arsenal] heap size [10156mb] compressed ordinary object pointers [true]

Page of 14 71 Page of 14 71

[2016-07-05 133859147][WARN ][env] [Arsenal] max file descriptors [4096] for elasticsearch process likely too low consider increasing to at least [65536]

[2016-07-05 133903526][INFO ][node] [Arsenal] initialized

[2016-07-05 133903526][INFO ][node] [Arsenal] starting

To13 test13 that13 Elascsearch13 is13 running13 correctly13 open13 the13 browser13 in13 the13 VNC13 client13 and13 run13 localhost920013

13

Install13 Kibana13 ELK13 recommends13 installing13 and13 running13 Kibana13 as13 root13 13

1 As13 root13 user13 create13 an13 installaon13 directory13 and13 untar13 the13 installaon13 file13

[rootelkforcase]$ cd home

[rootelkforcase]$ mkdir kibana

[rootelkforcase]$ cp homeibmcloudkibana-451-linux-x64targz homekibana

[rootelkforcase]$ cd kibana

[rootelkforcase]$ gunzip kibana-451-linux-x64targz

[rootelkforcase]$ untar kibana-451-linux-x64tar

2 Modify13 the13 manifestyml13 configuraon13 file13 Go13 to13 the13 kibana-shy‐451-shy‐linux-shy‐x6413 directory13

[rootelkforcase]$ cd homekibanakibana-451-linux-x64

3 Edit13 the13 manifestyml13 file13 so13 the13 contents13 are13

applicaons13

-shy‐ name13 kibana-shy‐in-shy‐bluemix13

memory13 128M13

host13 kibana-shy‐in-shy‐bluemix13

buildpack13 hcpsgithubcomcloudfoundry-shy‐communitynginx-shy‐buildpackgit13

Page of 15 71 Page of 15 71

4 Edit13 the13 configkibanayml13 file13 so13 the13 contents13 are13

Kibana is served by a back end server This controls which port to use

port 5601

The host to bind the server to

host 0000

If you are running kibana behind a proxy and want to mount it at a path

specify that path here The basePath cant end in a slash

serverbasePath

The maximum payload size in bytes on incoming server requests

servermaxPayloadBytes 1048576

The Elasticsearch instance to use for all your queries

elasticsearchurl httplocalhost9200

preserve_elasticsearch_host true will send the hostname specified in `elasticsearch` If you set it to false

then the host you use to connect to this Kibana instance will be sent

elasticsearchpreserveHost true

Kibana uses an index in Elasticsearch to store saved searches visualizations

and dashboards It will create a new index if it doesnt already exist

kibanaindex kibana

The default application to load

kibanadefaultAppId discover

If your Elasticsearch is protected with basic auth these are the user credentials

used by the Kibana server to perform maintenance on the kibana_index at startup Your

Kibana users will still need to authenticate with Elasticsearch (which is proxied through

the Kibana server)

Page of 16 71 Page of 16 71

elasticsearchusername user

elasticsearchpassword pass

SSL for outgoing requests from the Kibana Server to the browser (PEM formatted)

serversslcert pathtoyourservercrt

serversslkey pathtoyourserverkey

Optional setting to validate that your Elasticsearch backend uses the same key files (PEM formatted)

elasticsearchsslcert pathtoyourclientcrt

elasticsearchsslkey pathtoyourclientkey

If you need to provide a CA certificate for your Elasticsearch instance put

the path of the pem file here

elasticsearchsslca pathtoyourCApem

Set to false to have a complete disregard for the validity of the SSL

certificate

elasticsearchsslverify true

Time in milliseconds to wait for elasticsearch to respond to pings defaults to

request_timeout setting

elasticsearchpingTimeout 1500

Time in milliseconds to wait for responses from the back end or elasticsearch

This must be gt 0

elasticsearchrequestTimeout 30000

Time in milliseconds for Elasticsearch to wait for responses from shards

Set to 0 to disable

elasticsearchshardTimeout 0

Time in milliseconds to wait for Elasticsearch at Kibana startup before retrying

elasticsearchstartupTimeout 5000

elasticsearchstartupTimeout 30000

Page of 17 71 Page of 17 71

Set the path to where you would like the process id file to be created

pidfile varrunkibanapid

If you would like to send the log output to a file you can set the path below

loggingdest stdout

Set this to true to suppress all logging output

loggingsilent false

Set this to true to suppress all logging output except for error messages

loggingquiet false

Set this to true to log all events including system usage information and all requests

loggingverbose false

Kibana13 is13 now13 installed13

Start13 Kibana13 To13 start13 Kibana13 go13 to13 the13 bin13 directory13 and13 run13 the13 kibana13 executable13

[rootelkforcase]$

[rootelkforcase]$ cd homekibanakibana-451-linux-x64bin

[rootelkforcase]$ kibana amp

[1] 22283

[rootelkforcase] log [095900542] [info][status][pluginkibana] Status changed from uninitialized to green - Ready

log [095900600] [info][status][pluginelasticsearch] Status changed from uninitialized to yellow - Waiting for Elasticsearch

log [095900689] [info][status][pluginkbn_vislib_vis_types] Status changed from uninitialized to green - Ready

log [095900717] [info][status][pluginmarkdown_vis] Status changed from uninitialized to green - Ready

log [095900721] [info][status][pluginmetric_vis] Status changed from uninitialized to green - Ready

log [095900814] [info][status][pluginspyModes] Status changed from uninitialized to green - Ready

log [095900934] [info][status][pluginstatusPage] Status changed from uninitialized to green - Ready

Page of 18 71 Page of 18 71

log [095900941] [info][status][plugintable_vis] Status changed from uninitialized to green - Ready

log [095901067] [info][listening] Server running at http00005601

log [095906083] [info][status][pluginelasticsearch] Status changed from yellow to yellow - No existing Kibana index found

log [095908942] [info][status][pluginelasticsearch] Status changed from yellow to green - Kibana index ready

Kibana13 is13 now13 started13

Broadcast13 Bluemix13 app13 log13 records13 Bluemix13 apps13 (which13 are13 shown13 in13 the13 US13 South13 Bluemix13 region13 of13 the13 System13 Architecture13 diagram)13 will13 produce13 logs13 in13 a13 syslog13 rfc542413 format13 using13 a13 Bluemix13 tool13 called13 Loggregator13

For13 Logstash13 to13 receive13 these13 logs13 each13 Bluemix13 app13 needs13 to13 broadcast13 these13 logs13 To13 do13 that13 run13 the13 following13 commands13 on13 each13 Bluemix13 app13 using13 the13 public13 IP13 address13 of13 the13 ELK13 virtual13 server13 the13 ELK13 tag13 to13 trace13 the13 process13 and13 insert13 the13 name13 of13 each13 app13 in13 turn13

cf cups elk -l syslogltpublicIPaddress5000 cf bind-service ltmyappgt elk

cf restage ltmyappgt

Install13 Logstash13 To13 install13 Logstash13 as13 root13 user13 create13 a13 directory13 for13 the13 installaon13 and13 untar13 the13 installaon13 file13 there13

13 13 [rootelkforcase]$ cd home

[rootelkforcase]$ mkdir logstash

[rootelkforcase]$ cp homeibmcloudlogstash-233targz homelogstash

[rootelkforcase]$ cd logstash

[rootelkforcase]$ gunzip logstash-233targz

[rootelkforcase]$ untar logstash-233tar

Write13 Logstash13 configura0on13 Logstash13 requires13 a13 configuraon13 file13 to13 be13 wricen13 to13 receive13 transform13 and13 output13 the13 log13 records13 to13 Elascsearch13 This13 secon13 explains13 the13 basic13 requirements13 of13 processing13 Bluemix13 app13 log13 records13 into13 Elascsearch13 (more13 complex13 configuraons13 are13 detailed13 later13 in13 this13 document13 for13 the13 integraons13 with13 third-shy‐party13 tools)13

Create13 a13 conf13 file13 as13 root13 user13

[rootelkforcase]$ cd homelogstashlogstash-233

[rootelkforcase]$ mkdir config

Page of 19 71 Page of 19 71

[rootelkforcase]$ cd config

[rootelkforcase]$ touch loggregatorconf

Edit13 the13 loggregator13 file13 with13 the13 following13 contents13

13 Input13 secon13 defines13 which13 log13 records13 to13 receive13

13 port13 and13 type13 reflect13 method13 selected13 for13 Cloud13 Foundry13 to13 broadcast13 the13 records13

input

tcp

port =gt 5000

type =gt syslog

13 Filter13 secon13 ndash13 transforms13 the13 data13

filter

Add a field to identify the records as Cloud Foundry based

mutate

add_field =gt format =gt cf

13 Output13 secon13 ndash13 sends13 the13 data13 to13 its13 desnaon(s)13

output

13 Use13 the13 elascsearch13 plugin13 to13 send13 the13 data13 to13 the13 elascsearch13 instance13

13 elasticsearch

hosts =gt localhost

stdout codec =gt rubydebug

13 End13 of13 Config13

Save13 the13 file13 Verify13 the13 format13 of13 the13 file13 is13 acceptable13 by13 running13 the13 following13 commands13

[rootelkforcase]$ cd homelogstashlogstash-233

[rootelkforcase]$ binlogstash agent --configtest -f configloggregatorconf

Page of 20 71 Page of 20 71

Configuraon13 is13 OK13

Start13 Logstash13 Aaer13 you13 have13 wricen13 and13 verified13 the13 Logstash13 configuraon13 you13 can13 start13 Logstash13 to13 process13 records13 from13 the13 Bluemix13 apps13 (via13 Cloud13 Foundry)13 Run13 the13 following13 as13 root13 user13

[rootelkforcase]$ cd homelogstashlogstash-233

[rootelkforcase]$ binlogstash agent -f configloggregatorconf

Logstash13 is13 now13 started13

Monitor13 Logstash13 processing13 Once13 Logstash13 is13 running13 and13 log13 records13 are13 being13 produced13 by13 the13 Bluemix13 apps13 you13 can13 monitor13 progress13 in13 the13 same13 window13 that13 was13 used13 to13 start13 Logstash13 Output13 is13 similar13 to13 the13 following13

13

The13 log13 records13 being13 broadcast13 from13 the13 Bluemix13 apps13 are13 now13 sent13 to13 Elascsearch13 from13 the13 Logstash13 output13 configuraon13 Records13 are13 sent13 in13 real13 me13 If13 Logstash13 is13 not13 running13 while13 log13 records13 are13 being13 produced13 then13 these13 records13 will13 not13 be13 processed13 by13 Logstash13 and13 will13 not13 be13 stored13 in13 Elascsearch13

Page of 21 71 Page of 21 71

Use13 ELK13 The13 log13 records13 from13 the13 Bluemix13 apps13 are13 now13 being13 stored13 and13 indexed13 by13 the13 Elascsearch13 component13 This13 data13 can13 be13 accessed13 via13 the13 Kibana13 applicaon13

Kibana13 Discover13 page13 Using13 a13 web13 browser13 either13 directly13 from13 a13 client13 or13 using13 the13 VNC13 client13 access13 Kibana13 using13 the13 publicIPaddress13 of13 the13 ELK13 virtual13 server13

hcpltpublicIPaddressgt560113

No13 userpassword13 credenals13 are13 required13

Kibana13 defaults13 to13 a13 search13 of13 all13 records13 ()13 in13 the13 last13 1513 minutes13 and13 returns13 the13 results13

13

More13 complex13 searches13 are13 possible13 by13 changing13 the13 me13 filter13 and13 supplying13 search13 queries13

Change13 the13 0me13 filter13 To13 change13 the13 me13 ranges13 click13 the13 clock13 symbol13 in13 the13 top-shy‐right13 of13 the13 Kibana13 Discover13 page13

13

You13 will13 see13 the13 following13 opons13

13

Page of 22 71 Page of 22 71

A13 variety13 of13 quick13 me13 ranges13 can13 be13 selected13 or13 you13 can13 switch13 to13 relave13 or13 absolute13 me13 ranges13

Quick13 me13 ranges13 allow13 switching13 between13 last13 X13 minutes13 to13 last13 Y13 hours13 or13 Z13 days13 etc13

Relave13 me13 ranges13 allow13 more13 specific13 versions13 of13 the13 Quick13 opons13 from13 specific13 minuteshoursdays13 to13 now13

13

Absolute13 me13 ranges13 specify13 the13 start13 and13 end13 mes13 for13 the13 searches13

13

Custom13 search13 queries13 You13 can13 use13 the13 text13 window13 of13 the13 Discover13 page13 to13 search13 for13 matches13 within13 the13 log13 records13 You13 can13 use13 Apache13 Lucene13 or13 Elascsearch13 query13 DSL13 syntax13

13

Page of 23 71 Page of 23 71

Saved13 searches13 You13 can13 save13 any13 custom13 search13 syntax13 and13 choice13 of13 me13 range13 for13 future13 ease13 of13 use13 Any13 relave13 or13 absolute13 me13 frame13 is13 saved13 as13 part13 of13 the13 search13 13

To13 save13 a13 search13

1 Set13 up13 and13 run13 the13 required13 syntax13 and13 me13 frame13

13

2 Click13 the13 Save13 Search13 icon13

13

3 Type13 in13 a13 name13 for13 the13 saved13 search13

Page of 24 71 Page of 24 71

13

7 To13 delete13 or13 update13 saved13 searches13 click13 13 manage13 saved13 searches13

13

Use13 the13 edit13 icons13 to13 make13 any13 changes13

Page of 26 71 Page of 26 71

13

Build13 simple13 Kibana13 histograms13 Kibana13 allows13 you13 to13 build13 charts13 and13 dashboards13 based13 on13 the13 search13 capability13 offered13 by13 the13 Discover13 page13 Saved13 searches13 can13 be13 used13 as13 the13 basis13 for13 charts13 and13 dashboards13 built13 using13 those13 charts13 To13 build13 a13 simple13 histogram13

1 Select13 the13 Visualize13 page13

13

2 Select13 Ver0cal13 Bar13 Chart13

13

3 To13 use13 a13 saved13 search13 click13 From13 a13 saved13 search13

Page of 27 71 Page of 27 71

13

4 Select13 a13 previously13 saved13 search13

13

5 Select13 to13 edit13 the13 x-shy‐axis

6 Edit13 the13 fields13 with13 informaon13 about13 the13 aggregaon13 field13 interval13 and13 custom13 label13

Page of 28 71 Page of 28 71

13

Page of 29 71 Page of 29 71

7 Click13 the13 green13 arrow13 The13 histogram13 is13 created13 and13 run13 based13 on13 the13 saved13 search

13

8 To13 save13 the13 histogram13 click13 the13 Save13 Visualiza0on13 icon13 and13 add13 a13 tle13

13

13

9 Click13 Save13

10 To13 view13 the13 visualisaon13 click13 the13 Dashboard13 page13

Page of 30 71 Page of 30 71

13

11 Click13 the13 Add13 Visualiza0on13 icon13

13

12 Select13 the13 visiualizaon13 that13 you13 just13 created13

13

Page of 31 71 Page of 31 71

13

13 To13 save13 the13 dashboard13 click13 the13 Save13 Dashboard13 icon13

Page of 32 71 Page of 32 71

13

14 Add13 a13 tle13 and13 select13 whether13 to13 store13 the13 selected13 me13 frame13 with13 the13 dashboard13

13

15 Click13 Save13

16 You13 can13 edit13 the13 shape13 and13 number13 of13 charts13 and13 save13 the13 dashboard13

17 To13 re-shy‐run13 the13 dashboard13 select13 the13 Dashboard13 page13 and13 click13 the13 Load13 Saved13 Dashboard13 icon13

Page of 33 71 Page of 33 71

13

18 Select13 the13 saved13 dashboard13 from13 the13 list13

13

19 The13 dashboard13 runs13

13

Page of 34 71 Page of 34 71

Page of 35 71 Page of 35 71

Aler0ng13 with13 ELK13 ELK13 examines13 the13 log13 records13 in13 process13 to13 send13 alerts13 to13 third-shy‐party13 alerng13 event13 management13 and13 social13 media13 applicaons13 As13 shown13 previously13 in13 the13 ELK13 architecture13 the13 ELK13 strategy13 for13 alerng13 is13 to13 send13 alerts13 from13 the13 Logstash13 component13 directly13 Therefore13 log13 records13 and13 alerts13 are13 sent13 to13 both13 Elascsearch13 and13 third-shy‐party13 tools13 at13 the13 same13 instant13 using13 the13 output13 secon13 of13 the13 Logstash13 configuraon13 Any13 alerts13 are13 generated13 during13 the13 filter13 secon13 processing13 of13 the13 Logstash13 configuraon13

Create13 alerts13 in13 Logstash13 The13 Logstash13 filter13 secon13 is13 used13 to13 trigger13 alerts13 based13 on13 the13 log13 records13 being13 processed13 For13 example13 log13 records13 can13 be13 checked13 for13 errors13 warnings13 crashed13 or13 stopped13 messages13 and13 generate13 an13 alert13

Here13 is13 an13 example13 Logstash13 filter13 code13

13 if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

The13 specific13 tags13 are13 set13 against13 the13 log13 records13 to13 trigger13 the13 alert13 later13 in13 the13 configuraon13 in13 the13 output13 secon13

Control13 the13 number13 of13 generated13 alerts13 using13 the13 Logstash13 throcle13 plugin13 so13 that13 either13 a13 lot13 of13 instances13 in13 a13 fixed13 me13 period13 are13 required13 before13 an13 alert13 is13 sent13 or13 a13 specific13 number13 of13 instances13 triggers13 the13 alert13

Page of 36 71 Page of 36 71

Edit13 the13 plugin13 parameters13 to13 meet13 system13 requirements13

13 13 13 13 13 if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 10

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle13

after_count =gt 20

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 30

period =gt 1200

key =gt servicename_alarm_warning

Page of 37 71 Page of 37 71

add_tag =gt servicename_throttled

13 13 13 13 13 13 13 13 13

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 40

period =gt 1500

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

Enrich13 alerts13 with13 message13 detail13 Some13 of13 the13 third-shy‐party13 integraons13 benefit13 from13 having13 components13 of13 the13 raw13 log13 record13 sent13 in13 addion13 to13 the13 raw13 record13 or13 fields13 in13 the13 record13 edited13 transformed13 or13 translated13

Grok13 Logstash13 allows13 the13 log13 record13 to13 be13 broken13 up13 into13 logical13 secons13 and13 each13 secon13 stored13 in13 a13 new13 parameter13 using13 the13 grok13 parameter13 in13 the13 filter13 secon13 Here13 is13 an13 example13 grok13

13 grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

To13 build13 the13 grok13 statement13 Logstash13 offers13 an13 online13 tool13 at13 hcpsgrokdebugherokuappcom13

Translate13 If13 specific13 fields13 in13 the13 log13 record13 are13 coded13 (for13 example13 the13 Bluemix13 app13 name13 is13 oaen13 not13 sent13 in13 the13 syslog13 messages13 instead13 the13 GUID13 is13 sent13 which13 is13 not13 useful)13 then13 the13 filter13 secon13 translate13 plugin13 can13 be13 used13 to13 add13 new13 fields13 by13 holding13 the13 decoded13 versions13 of13 those13 values13

13 translate

field =gt data1

destination =gt data1

override =gt true

Page of 38 71 Page of 38 71

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

dc50f6e4-c052-436a-bf27-95ba392c4228 monverifyservicemanagement

0c677e82-5bfc-4e38-9155-e416249e3ee6 ms-catalog-postpyretic-pomiculture

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

However13 the13 translate13 plugin13 is13 not13 included13 with13 the13 Logstash13 installaon13 In13 the13 next13 secon13 I13 will13 13 describe13 how13 to13 add13 new13 plugins13 to13 the13 Logstash13 instance13

Add13 custom13 plugins13 to13 Logstash13 As13 an13 open13 source13 tool13 Logstash13 encourages13 users13 to13 develop13 content13 for13 all13 users13 to13 share13 Some13 of13 the13 plugins13 that13 were13 created13 by13 our13 community13 include13 a13 translaon13 plugin13 an13 output13 Slack13 plugin13 and13 an13 output13 syslog13 plugin13 Letrsquos13 look13 at13 how13 to13 add13 these13 plugins13 to13 your13 environment13

Filter13 translate13 plugin13 To13 add13 the13 translate13 plugin13

1 Download13 the13 plugin13

2 Copy13 the13 plugin13 file13 logstash-shy‐filter-shy‐translate-shy‐214gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-filter-translate 214

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Output13 Slack13 plugin13 To13 add13 the13 Slack13 plugin13

Page of 39 71 Page of 39 71

1 Download13 the13 plugin13 13

2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐slack-shy‐014gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-output-slack 014

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Output13 syslog13 plugin13 To13 add13 the13 syslog13 plugin13

1 Download13 the13 plugin13 13

2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐syslog-shy‐214gem13 to13 the13 ELK13 Virtual13 Server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-output-syslog 214

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Integrate13 Logstash13 with13 Slack13 To13 send13 generated13 alerts13 to13 Slack13 complete13 these13 steps13

1 Create13 a13 Slack13 integraon13 13

2 Use13 the13 Slack13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 Slack13 integraon13

113 Login13 to13 Slack13 and13 go13 to13 the13 Apps13 page13 13

Page of 40 71 Page of 40 71

13

313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13

13

A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13

13

Page of 42 71 Page of 42 71

The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13

13

The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13

13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

slack

url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5

channel =gt cloud-operations-mon

username =gt nick_cawood

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13

Page of 43 71 Page of 43 71

Page of 44 71 Page of 44 71

In13 Slack13 the13 alerts13 look13 like13 this13

13

Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13

1 Create13 a13 PagerDuty13 integraon13 13

2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13

1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13

13

2 Click13 Add13 New13 Service13

13

313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13

Page of 45 71 Page of 45 71

13

413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13

513 Click13 Add13 Service13

13

The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13

Page of 46 71 Page of 46 71

13

The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt be7eb1ea9c224018923e818d3e891d0b

incident_key =gt logstashhosttype

Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13

Page of 47 71 Page of 47 71

13

Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13

1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13

2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13

3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13

4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13

13 13 http

url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo

http_method =gt post

format =gt json

For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13

13 13 syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

Page of 48 71 Page of 48 71

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13

13

The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13

if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2

This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13

Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13

Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13

Logstash13 severity Shown13 in13 Omnibus13 as

alert Indeterminate

crical Warning

debug 7

emergency Clear

error Minor

informaonal 6

noce Crical

warning Major

Page of 49 71 Page of 49 71

13

Page of 50 71 Page of 50 71

Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13

input

tcp

port =gt 5000

type =gt syslog

filter

mutate

add_field =gt format =gt cf

grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

translate

field =gt data1

destination =gt data1

override =gt true

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

=== Check Alarm Criteria ===

Standard Key Words

if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

=== Alarm Exclusions ===

if [message] =~ (i)sometexthere

mutate remove_tag =gt servicename_alarm

Throttle alarms so we do not hit PD rate limits

Page of 52 71 Page of 52 71

if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 5

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle

after_count =gt 5

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 5

period =gt 1200

key =gt servicename_alarm_warning

add_tag =gt servicename_throttled

Page of 53 71 Page of 53 71

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

output

elasticsearch

hosts =gt localhost

stdout codec =gt rubydebug

if servicename_alarm_err in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

Page of 54 71 Page of 54 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 55 71 Page of 55 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

Page of 56 71 Page of 56 71

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert err message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_error in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

Page of 57 71 Page of 57 71

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 58 71 Page of 58 71

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 59 71 Page of 59 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert error message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

Page of 60 71 Page of 60 71

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

Page of 61 71 Page of 61 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 62 71 Page of 62 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt x

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert fatal message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

Page of 63 71 Page of 63 71

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

Page of 64 71 Page of 64 71

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 65 71 Page of 65 71

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt heavy_exclamation_mark

Page of 66 71 Page of 66 71

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert WARNING message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 67 71 Page of 67 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

Page of 68 71 Page of 68 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 69 71 Page of 69 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

Page of 70 71 Page of 70 71

Page of 71 71 Page of 71 71

Page 10: How to integrate the Elasticsearch Logstash Kibana (ELK ... · How to integrate the Elasticsearch Logstash Kibana (ELK) log analytics stack into IBM Bluemix Nick Cawood IBM Cloud

4 Name13 the13 virtual13 server13

5 Make13 sure13 Assign13 public13 IP13 address13 is13 checked13

13

6 Click13 Add13 Key13 to13 add13 the13 SSH13 key13 you13 created13 previously13 Copy13 the13 contents13 of13 the13 public13 key13 into13 the13 Public13 Key13 to13 import13 field13

Page of 10 71 Page of 10 71

13

7 Click13 OK13 Then13 click13 CREATE13

The13 virtual13 server13 is13 provisioned13 and13 added13 to13 the13 Bluemix13 dashboard13 under13 VIRTUAL13 SERVERS13

Page of 11 71 Page of 11 71

13

Set13 up13 the13 virtual13 server13

Log13 on13 to13 the13 virtual13 server13 Now13 you13 can13 log13 on13 to13 the13 virtual13 server13 and13 add13 new13 users13 and13 programs13 to13 make13 it13 easier13 to13 access13 the13 virtual13 server13

To13 access13 the13 server13 go13 to13 the13 UNIX13 or13 Linux13 window13 that13 you13 used13 to13 create13 the13 SSH13 key13 In13 the13 directory13 where13 the13 ltuniquekeynamegt13 files13 exist13 run13 the13 following13 command13 referencing13 the13 SSH13 keyname13 the13 public13 IP13 address13 Bluemix13 allocated13 for13 the13 virtual13 server13 and13 the13 password13 for13 the13 SSH13 key13

[userserver ~]$ ssh -i ltuniquekeynamegt ibmcloudltpublicIPaddressgt

Enter13 the13 passphrase13 for13 the13 key13 ltuniquekeynamegt13

Last login Tue Jul 5 125546 2016 from 108-210-96-231lightspeedrlghncsbcglobalnet

[ibmcloudelkforcase ~]$

ibmcloud13 is13 the13 default13 user13 for13 the13 Bluemix13 virtual13 servers13 and13 allows13 SSH13 to13 access13 the13 server13

Acces13 the13 root13 user13 To13 access13 the13 root13 user13 log13 on13 as13 ibmcloud13 user13 and13 run13 the13 following13 command13

[ibmcloudelkforcase ~]$ sudo bash

[rootelkforcase ibmcloud]

The13 root13 user13 can13 be13 used13 to13 add13 programs13 to13 the13 server13 and13 create13 and13 edit13 users13 as13 required13 to13 access13 applicaons13 installed13 on13 the13 virtual13 server13

Page of 12 71 Page of 12 71

Copy13 files13 to13 the13 virtual13 server13 To13 move13 installaon13 files13 to13 the13 server13 go13 to13 the13 UNIX13 or13 Linux13 window13 you13 used13 to13 create13 the13 SSH13 key13 and13 the13 directory13 where13 the13 ltuniquekeynamegt13 files13 exist13 Move13 the13 installaon13 files13 to13 the13 same13 locaon13

Run13 the13 following13 command13 referencing13 the13 SSH13 keyname13 the13 public13 IP13 address13 Bluemix13 allocated13 for13 the13 virtual13 server13 and13 the13 password13 for13 the13 SSH13 key13

[userserver ~]$ scp ndashi ltuniquekeynamegt ltINSTALLATIONFILENAMEgt ibmcloudltpublicIPaddressgthomeibmcloud

Enter13 the13 passphrase13 for13 key13 cloudkey313

This13 copies13 the13 file13 using13 the13 ibmcloud13 user13 to13 the13 homeibmcloud13 directory13 on13 the13 virtual13 server13

Install13 and13 access13 the13 virtual13 server13 via13 VNC13 server13 Using13 the13 ssh13 command13 to13 access13 the13 virtual13 server13 is13 not13 the13 most13 stable13 method13 and13 does13 not13 offer13 X-shy‐Windows13 support13 Therefore13 we13 suggest13 that13 you13 use13 a13 VNC13 client13 to13 access13 the13 virtual13 server13

Installing13 a13 VNC13 client13 should13 be13 done13 as13 the13 root13 user13 Use13 the13 yum13 command13 as13 follows13

[ibmcloudelkforcase ~]$ yum groupinstall GNOME Desktop

[ibmcloudelkforcase ~]$ yum install tigervnc-server

These13 commands13 first13 install13 a13 desktop13 style13 OS13 user13 interface13 and13 then13 a13 VNC13 client13 to13 remotely13 access13 the13 desktop13 style13 OS13

Then13 as13 either13 the13 root13 user13 or13 another13 user13 (example13 below13 uses13 elk)13 run13 the13 following13 command13 to13 create13 the13 VNC13 client13 connecon13

[elkelkforcase ~]$ vncserver

New elkforcase2 (elk) desktop is elkforcase2

Starting applications specified in homeelkvncxstartup

Log file is homeelkvncelkforcase2log

[elkelkforcase ~]$

Install13 a13 VNC13 client13 on13 the13 client13 machine13 used13 to13 access13 the13 Internet13 and13 (in13 the13 example13 above)13 use13 the13 command13 ltpublicIPadddressgt213 and13 the13 password13 for13 ELK13 to13 access13 the13 virtual13 server13

Using13 a13 VNC13 client13 means13 that13 you13 can13 leave13 long13 duraon13 installaons13 running13 without13 fear13 of13 the13 terminal13 losing13 connecon13 and13 the13 installaons13 failing13

Install13 the13 ELK13 stack13 Download13 the13 source13 soaware13 code13 for13 the13 ELK13 stack13 from13 the13 ELK13 website13 13

At13 the13 me13 of13 wring13 the13 downloaded13 soaware13 files13 were13

Page of 13 71 Page of 13 71

13

Transfer13 the13 files13 to13 the13 virtual13 server13 (using13 the13 scp13 method13 detailed13 above)13

Install13 Elas0csearch13 As13 a13 root13 user13 create13 a13 non-shy‐root13 user13 with13 the13 adduser13 command13

Create13 a13 directory13 to13 install13 Elascsearch13 into13 and13 untar13 the13 contents13 of13 the13 Elascsearch13 install13 file13 into13 that13 directory13

[elkelkforcase]$ cd home

[elkelkforcase]$ mkdir elasticsearch

[elkelkforcase]$ cp homeibmcloudelasticsearch-233targz homeelasticsearch

[elkelkforcase]$ cd elasticsearch

[elkelkforcase]$ gunzip elasticsearch-233targz

[elkelkforcase]$ untar elasticsearch-233tar

Elascsearch13 is13 now13 installed13

Start13 Elas0csearch13 To13 start13 Elascsearch13 go13 to13 the13 bin13 directory13 as13 the13 non-shy‐root13 user13 when13 you13 installed13 it13 and13 run13 the13 elascsearch13 executable13

[elkelkforcase]$ cd homeelasticsearchelasticsearch-233bin

[elkelkforcase]$ elasticsearch amp

[2] 22211

[elkelkforcase]$ [2016-07-05 133857730][INFO ][node] [Arsenal] version[233] pid[22211] build[218bdf12016-05-17T154004Z]

[2016-07-05 133857740][INFO ][node] [Arsenal] initializing

[2016-07-05 133858984][INFO ][plugins] [Arsenal] modules [reindex lang-expression lang-groovy] plugins [] sites []

[2016-07-05 133859146][INFO ][env] [Arsenal] using [1] data paths mounts [[ (rootfs)]] net usable_space [155gb] net total_space [199gb] spins [unknown] types [rootfs]

[2016-07-05 133859146][INFO ][env] [Arsenal] heap size [10156mb] compressed ordinary object pointers [true]

Page of 14 71 Page of 14 71

[2016-07-05 133859147][WARN ][env] [Arsenal] max file descriptors [4096] for elasticsearch process likely too low consider increasing to at least [65536]

[2016-07-05 133903526][INFO ][node] [Arsenal] initialized

[2016-07-05 133903526][INFO ][node] [Arsenal] starting

To13 test13 that13 Elascsearch13 is13 running13 correctly13 open13 the13 browser13 in13 the13 VNC13 client13 and13 run13 localhost920013

13

Install13 Kibana13 ELK13 recommends13 installing13 and13 running13 Kibana13 as13 root13 13

1 As13 root13 user13 create13 an13 installaon13 directory13 and13 untar13 the13 installaon13 file13

[rootelkforcase]$ cd home

[rootelkforcase]$ mkdir kibana

[rootelkforcase]$ cp homeibmcloudkibana-451-linux-x64targz homekibana

[rootelkforcase]$ cd kibana

[rootelkforcase]$ gunzip kibana-451-linux-x64targz

[rootelkforcase]$ untar kibana-451-linux-x64tar

2 Modify13 the13 manifestyml13 configuraon13 file13 Go13 to13 the13 kibana-shy‐451-shy‐linux-shy‐x6413 directory13

[rootelkforcase]$ cd homekibanakibana-451-linux-x64

3 Edit13 the13 manifestyml13 file13 so13 the13 contents13 are13

applicaons13

-shy‐ name13 kibana-shy‐in-shy‐bluemix13

memory13 128M13

host13 kibana-shy‐in-shy‐bluemix13

buildpack13 hcpsgithubcomcloudfoundry-shy‐communitynginx-shy‐buildpackgit13

Page of 15 71 Page of 15 71

4 Edit13 the13 configkibanayml13 file13 so13 the13 contents13 are13

Kibana is served by a back end server This controls which port to use

port 5601

The host to bind the server to

host 0000

If you are running kibana behind a proxy and want to mount it at a path

specify that path here The basePath cant end in a slash

serverbasePath

The maximum payload size in bytes on incoming server requests

servermaxPayloadBytes 1048576

The Elasticsearch instance to use for all your queries

elasticsearchurl httplocalhost9200

preserve_elasticsearch_host true will send the hostname specified in `elasticsearch` If you set it to false

then the host you use to connect to this Kibana instance will be sent

elasticsearchpreserveHost true

Kibana uses an index in Elasticsearch to store saved searches visualizations

and dashboards It will create a new index if it doesnt already exist

kibanaindex kibana

The default application to load

kibanadefaultAppId discover

If your Elasticsearch is protected with basic auth these are the user credentials

used by the Kibana server to perform maintenance on the kibana_index at startup Your

Kibana users will still need to authenticate with Elasticsearch (which is proxied through

the Kibana server)

Page of 16 71 Page of 16 71

elasticsearchusername user

elasticsearchpassword pass

SSL for outgoing requests from the Kibana Server to the browser (PEM formatted)

serversslcert pathtoyourservercrt

serversslkey pathtoyourserverkey

Optional setting to validate that your Elasticsearch backend uses the same key files (PEM formatted)

elasticsearchsslcert pathtoyourclientcrt

elasticsearchsslkey pathtoyourclientkey

If you need to provide a CA certificate for your Elasticsearch instance put

the path of the pem file here

elasticsearchsslca pathtoyourCApem

Set to false to have a complete disregard for the validity of the SSL

certificate

elasticsearchsslverify true

Time in milliseconds to wait for elasticsearch to respond to pings defaults to

request_timeout setting

elasticsearchpingTimeout 1500

Time in milliseconds to wait for responses from the back end or elasticsearch

This must be gt 0

elasticsearchrequestTimeout 30000

Time in milliseconds for Elasticsearch to wait for responses from shards

Set to 0 to disable

elasticsearchshardTimeout 0

Time in milliseconds to wait for Elasticsearch at Kibana startup before retrying

elasticsearchstartupTimeout 5000

elasticsearchstartupTimeout 30000

Page of 17 71 Page of 17 71

Set the path to where you would like the process id file to be created

pidfile varrunkibanapid

If you would like to send the log output to a file you can set the path below

loggingdest stdout

Set this to true to suppress all logging output

loggingsilent false

Set this to true to suppress all logging output except for error messages

loggingquiet false

Set this to true to log all events including system usage information and all requests

loggingverbose false

Kibana13 is13 now13 installed13

Start13 Kibana13 To13 start13 Kibana13 go13 to13 the13 bin13 directory13 and13 run13 the13 kibana13 executable13

[rootelkforcase]$

[rootelkforcase]$ cd homekibanakibana-451-linux-x64bin

[rootelkforcase]$ kibana amp

[1] 22283

[rootelkforcase] log [095900542] [info][status][pluginkibana] Status changed from uninitialized to green - Ready

log [095900600] [info][status][pluginelasticsearch] Status changed from uninitialized to yellow - Waiting for Elasticsearch

log [095900689] [info][status][pluginkbn_vislib_vis_types] Status changed from uninitialized to green - Ready

log [095900717] [info][status][pluginmarkdown_vis] Status changed from uninitialized to green - Ready

log [095900721] [info][status][pluginmetric_vis] Status changed from uninitialized to green - Ready

log [095900814] [info][status][pluginspyModes] Status changed from uninitialized to green - Ready

log [095900934] [info][status][pluginstatusPage] Status changed from uninitialized to green - Ready

Page of 18 71 Page of 18 71

log [095900941] [info][status][plugintable_vis] Status changed from uninitialized to green - Ready

log [095901067] [info][listening] Server running at http00005601

log [095906083] [info][status][pluginelasticsearch] Status changed from yellow to yellow - No existing Kibana index found

log [095908942] [info][status][pluginelasticsearch] Status changed from yellow to green - Kibana index ready

Kibana13 is13 now13 started13

Broadcast13 Bluemix13 app13 log13 records13 Bluemix13 apps13 (which13 are13 shown13 in13 the13 US13 South13 Bluemix13 region13 of13 the13 System13 Architecture13 diagram)13 will13 produce13 logs13 in13 a13 syslog13 rfc542413 format13 using13 a13 Bluemix13 tool13 called13 Loggregator13

For13 Logstash13 to13 receive13 these13 logs13 each13 Bluemix13 app13 needs13 to13 broadcast13 these13 logs13 To13 do13 that13 run13 the13 following13 commands13 on13 each13 Bluemix13 app13 using13 the13 public13 IP13 address13 of13 the13 ELK13 virtual13 server13 the13 ELK13 tag13 to13 trace13 the13 process13 and13 insert13 the13 name13 of13 each13 app13 in13 turn13

cf cups elk -l syslogltpublicIPaddress5000 cf bind-service ltmyappgt elk

cf restage ltmyappgt

Install13 Logstash13 To13 install13 Logstash13 as13 root13 user13 create13 a13 directory13 for13 the13 installaon13 and13 untar13 the13 installaon13 file13 there13

13 13 [rootelkforcase]$ cd home

[rootelkforcase]$ mkdir logstash

[rootelkforcase]$ cp homeibmcloudlogstash-233targz homelogstash

[rootelkforcase]$ cd logstash

[rootelkforcase]$ gunzip logstash-233targz

[rootelkforcase]$ untar logstash-233tar

Write13 Logstash13 configura0on13 Logstash13 requires13 a13 configuraon13 file13 to13 be13 wricen13 to13 receive13 transform13 and13 output13 the13 log13 records13 to13 Elascsearch13 This13 secon13 explains13 the13 basic13 requirements13 of13 processing13 Bluemix13 app13 log13 records13 into13 Elascsearch13 (more13 complex13 configuraons13 are13 detailed13 later13 in13 this13 document13 for13 the13 integraons13 with13 third-shy‐party13 tools)13

Create13 a13 conf13 file13 as13 root13 user13

[rootelkforcase]$ cd homelogstashlogstash-233

[rootelkforcase]$ mkdir config

Page of 19 71 Page of 19 71

[rootelkforcase]$ cd config

[rootelkforcase]$ touch loggregatorconf

Edit13 the13 loggregator13 file13 with13 the13 following13 contents13

13 Input13 secon13 defines13 which13 log13 records13 to13 receive13

13 port13 and13 type13 reflect13 method13 selected13 for13 Cloud13 Foundry13 to13 broadcast13 the13 records13

input

tcp

port =gt 5000

type =gt syslog

13 Filter13 secon13 ndash13 transforms13 the13 data13

filter

Add a field to identify the records as Cloud Foundry based

mutate

add_field =gt format =gt cf

13 Output13 secon13 ndash13 sends13 the13 data13 to13 its13 desnaon(s)13

output

13 Use13 the13 elascsearch13 plugin13 to13 send13 the13 data13 to13 the13 elascsearch13 instance13

13 elasticsearch

hosts =gt localhost

stdout codec =gt rubydebug

13 End13 of13 Config13

Save13 the13 file13 Verify13 the13 format13 of13 the13 file13 is13 acceptable13 by13 running13 the13 following13 commands13

[rootelkforcase]$ cd homelogstashlogstash-233

[rootelkforcase]$ binlogstash agent --configtest -f configloggregatorconf

Page of 20 71 Page of 20 71

Configuraon13 is13 OK13

Start13 Logstash13 Aaer13 you13 have13 wricen13 and13 verified13 the13 Logstash13 configuraon13 you13 can13 start13 Logstash13 to13 process13 records13 from13 the13 Bluemix13 apps13 (via13 Cloud13 Foundry)13 Run13 the13 following13 as13 root13 user13

[rootelkforcase]$ cd homelogstashlogstash-233

[rootelkforcase]$ binlogstash agent -f configloggregatorconf

Logstash13 is13 now13 started13

Monitor13 Logstash13 processing13 Once13 Logstash13 is13 running13 and13 log13 records13 are13 being13 produced13 by13 the13 Bluemix13 apps13 you13 can13 monitor13 progress13 in13 the13 same13 window13 that13 was13 used13 to13 start13 Logstash13 Output13 is13 similar13 to13 the13 following13

13

The13 log13 records13 being13 broadcast13 from13 the13 Bluemix13 apps13 are13 now13 sent13 to13 Elascsearch13 from13 the13 Logstash13 output13 configuraon13 Records13 are13 sent13 in13 real13 me13 If13 Logstash13 is13 not13 running13 while13 log13 records13 are13 being13 produced13 then13 these13 records13 will13 not13 be13 processed13 by13 Logstash13 and13 will13 not13 be13 stored13 in13 Elascsearch13

Page of 21 71 Page of 21 71

Use13 ELK13 The13 log13 records13 from13 the13 Bluemix13 apps13 are13 now13 being13 stored13 and13 indexed13 by13 the13 Elascsearch13 component13 This13 data13 can13 be13 accessed13 via13 the13 Kibana13 applicaon13

Kibana13 Discover13 page13 Using13 a13 web13 browser13 either13 directly13 from13 a13 client13 or13 using13 the13 VNC13 client13 access13 Kibana13 using13 the13 publicIPaddress13 of13 the13 ELK13 virtual13 server13

hcpltpublicIPaddressgt560113

No13 userpassword13 credenals13 are13 required13

Kibana13 defaults13 to13 a13 search13 of13 all13 records13 ()13 in13 the13 last13 1513 minutes13 and13 returns13 the13 results13

13

More13 complex13 searches13 are13 possible13 by13 changing13 the13 me13 filter13 and13 supplying13 search13 queries13

Change13 the13 0me13 filter13 To13 change13 the13 me13 ranges13 click13 the13 clock13 symbol13 in13 the13 top-shy‐right13 of13 the13 Kibana13 Discover13 page13

13

You13 will13 see13 the13 following13 opons13

13

Page of 22 71 Page of 22 71

A13 variety13 of13 quick13 me13 ranges13 can13 be13 selected13 or13 you13 can13 switch13 to13 relave13 or13 absolute13 me13 ranges13

Quick13 me13 ranges13 allow13 switching13 between13 last13 X13 minutes13 to13 last13 Y13 hours13 or13 Z13 days13 etc13

Relave13 me13 ranges13 allow13 more13 specific13 versions13 of13 the13 Quick13 opons13 from13 specific13 minuteshoursdays13 to13 now13

13

Absolute13 me13 ranges13 specify13 the13 start13 and13 end13 mes13 for13 the13 searches13

13

Custom13 search13 queries13 You13 can13 use13 the13 text13 window13 of13 the13 Discover13 page13 to13 search13 for13 matches13 within13 the13 log13 records13 You13 can13 use13 Apache13 Lucene13 or13 Elascsearch13 query13 DSL13 syntax13

13

Page of 23 71 Page of 23 71

Saved13 searches13 You13 can13 save13 any13 custom13 search13 syntax13 and13 choice13 of13 me13 range13 for13 future13 ease13 of13 use13 Any13 relave13 or13 absolute13 me13 frame13 is13 saved13 as13 part13 of13 the13 search13 13

To13 save13 a13 search13

1 Set13 up13 and13 run13 the13 required13 syntax13 and13 me13 frame13

13

2 Click13 the13 Save13 Search13 icon13

13

3 Type13 in13 a13 name13 for13 the13 saved13 search13

Page of 24 71 Page of 24 71

13

7 To13 delete13 or13 update13 saved13 searches13 click13 13 manage13 saved13 searches13

13

Use13 the13 edit13 icons13 to13 make13 any13 changes13

Page of 26 71 Page of 26 71

13

Build13 simple13 Kibana13 histograms13 Kibana13 allows13 you13 to13 build13 charts13 and13 dashboards13 based13 on13 the13 search13 capability13 offered13 by13 the13 Discover13 page13 Saved13 searches13 can13 be13 used13 as13 the13 basis13 for13 charts13 and13 dashboards13 built13 using13 those13 charts13 To13 build13 a13 simple13 histogram13

1 Select13 the13 Visualize13 page13

13

2 Select13 Ver0cal13 Bar13 Chart13

13

3 To13 use13 a13 saved13 search13 click13 From13 a13 saved13 search13

Page of 27 71 Page of 27 71

13

4 Select13 a13 previously13 saved13 search13

13

5 Select13 to13 edit13 the13 x-shy‐axis

6 Edit13 the13 fields13 with13 informaon13 about13 the13 aggregaon13 field13 interval13 and13 custom13 label13

Page of 28 71 Page of 28 71

13

Page of 29 71 Page of 29 71

7 Click13 the13 green13 arrow13 The13 histogram13 is13 created13 and13 run13 based13 on13 the13 saved13 search

13

8 To13 save13 the13 histogram13 click13 the13 Save13 Visualiza0on13 icon13 and13 add13 a13 tle13

13

13

9 Click13 Save13

10 To13 view13 the13 visualisaon13 click13 the13 Dashboard13 page13

Page of 30 71 Page of 30 71

13

11 Click13 the13 Add13 Visualiza0on13 icon13

13

12 Select13 the13 visiualizaon13 that13 you13 just13 created13

13

Page of 31 71 Page of 31 71

13

13 To13 save13 the13 dashboard13 click13 the13 Save13 Dashboard13 icon13

Page of 32 71 Page of 32 71

13

14 Add13 a13 tle13 and13 select13 whether13 to13 store13 the13 selected13 me13 frame13 with13 the13 dashboard13

13

15 Click13 Save13

16 You13 can13 edit13 the13 shape13 and13 number13 of13 charts13 and13 save13 the13 dashboard13

17 To13 re-shy‐run13 the13 dashboard13 select13 the13 Dashboard13 page13 and13 click13 the13 Load13 Saved13 Dashboard13 icon13

Page of 33 71 Page of 33 71

13

18 Select13 the13 saved13 dashboard13 from13 the13 list13

13

19 The13 dashboard13 runs13

13

Page of 34 71 Page of 34 71

Page of 35 71 Page of 35 71

Aler0ng13 with13 ELK13 ELK13 examines13 the13 log13 records13 in13 process13 to13 send13 alerts13 to13 third-shy‐party13 alerng13 event13 management13 and13 social13 media13 applicaons13 As13 shown13 previously13 in13 the13 ELK13 architecture13 the13 ELK13 strategy13 for13 alerng13 is13 to13 send13 alerts13 from13 the13 Logstash13 component13 directly13 Therefore13 log13 records13 and13 alerts13 are13 sent13 to13 both13 Elascsearch13 and13 third-shy‐party13 tools13 at13 the13 same13 instant13 using13 the13 output13 secon13 of13 the13 Logstash13 configuraon13 Any13 alerts13 are13 generated13 during13 the13 filter13 secon13 processing13 of13 the13 Logstash13 configuraon13

Create13 alerts13 in13 Logstash13 The13 Logstash13 filter13 secon13 is13 used13 to13 trigger13 alerts13 based13 on13 the13 log13 records13 being13 processed13 For13 example13 log13 records13 can13 be13 checked13 for13 errors13 warnings13 crashed13 or13 stopped13 messages13 and13 generate13 an13 alert13

Here13 is13 an13 example13 Logstash13 filter13 code13

13 if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

The13 specific13 tags13 are13 set13 against13 the13 log13 records13 to13 trigger13 the13 alert13 later13 in13 the13 configuraon13 in13 the13 output13 secon13

Control13 the13 number13 of13 generated13 alerts13 using13 the13 Logstash13 throcle13 plugin13 so13 that13 either13 a13 lot13 of13 instances13 in13 a13 fixed13 me13 period13 are13 required13 before13 an13 alert13 is13 sent13 or13 a13 specific13 number13 of13 instances13 triggers13 the13 alert13

Page of 36 71 Page of 36 71

Edit13 the13 plugin13 parameters13 to13 meet13 system13 requirements13

13 13 13 13 13 if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 10

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle13

after_count =gt 20

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 30

period =gt 1200

key =gt servicename_alarm_warning

Page of 37 71 Page of 37 71

add_tag =gt servicename_throttled

13 13 13 13 13 13 13 13 13

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 40

period =gt 1500

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

Enrich13 alerts13 with13 message13 detail13 Some13 of13 the13 third-shy‐party13 integraons13 benefit13 from13 having13 components13 of13 the13 raw13 log13 record13 sent13 in13 addion13 to13 the13 raw13 record13 or13 fields13 in13 the13 record13 edited13 transformed13 or13 translated13

Grok13 Logstash13 allows13 the13 log13 record13 to13 be13 broken13 up13 into13 logical13 secons13 and13 each13 secon13 stored13 in13 a13 new13 parameter13 using13 the13 grok13 parameter13 in13 the13 filter13 secon13 Here13 is13 an13 example13 grok13

13 grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

To13 build13 the13 grok13 statement13 Logstash13 offers13 an13 online13 tool13 at13 hcpsgrokdebugherokuappcom13

Translate13 If13 specific13 fields13 in13 the13 log13 record13 are13 coded13 (for13 example13 the13 Bluemix13 app13 name13 is13 oaen13 not13 sent13 in13 the13 syslog13 messages13 instead13 the13 GUID13 is13 sent13 which13 is13 not13 useful)13 then13 the13 filter13 secon13 translate13 plugin13 can13 be13 used13 to13 add13 new13 fields13 by13 holding13 the13 decoded13 versions13 of13 those13 values13

13 translate

field =gt data1

destination =gt data1

override =gt true

Page of 38 71 Page of 38 71

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

dc50f6e4-c052-436a-bf27-95ba392c4228 monverifyservicemanagement

0c677e82-5bfc-4e38-9155-e416249e3ee6 ms-catalog-postpyretic-pomiculture

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

However13 the13 translate13 plugin13 is13 not13 included13 with13 the13 Logstash13 installaon13 In13 the13 next13 secon13 I13 will13 13 describe13 how13 to13 add13 new13 plugins13 to13 the13 Logstash13 instance13

Add13 custom13 plugins13 to13 Logstash13 As13 an13 open13 source13 tool13 Logstash13 encourages13 users13 to13 develop13 content13 for13 all13 users13 to13 share13 Some13 of13 the13 plugins13 that13 were13 created13 by13 our13 community13 include13 a13 translaon13 plugin13 an13 output13 Slack13 plugin13 and13 an13 output13 syslog13 plugin13 Letrsquos13 look13 at13 how13 to13 add13 these13 plugins13 to13 your13 environment13

Filter13 translate13 plugin13 To13 add13 the13 translate13 plugin13

1 Download13 the13 plugin13

2 Copy13 the13 plugin13 file13 logstash-shy‐filter-shy‐translate-shy‐214gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-filter-translate 214

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Output13 Slack13 plugin13 To13 add13 the13 Slack13 plugin13

Page of 39 71 Page of 39 71

1 Download13 the13 plugin13 13

2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐slack-shy‐014gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-output-slack 014

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Output13 syslog13 plugin13 To13 add13 the13 syslog13 plugin13

1 Download13 the13 plugin13 13

2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐syslog-shy‐214gem13 to13 the13 ELK13 Virtual13 Server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-output-syslog 214

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Integrate13 Logstash13 with13 Slack13 To13 send13 generated13 alerts13 to13 Slack13 complete13 these13 steps13

1 Create13 a13 Slack13 integraon13 13

2 Use13 the13 Slack13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 Slack13 integraon13

113 Login13 to13 Slack13 and13 go13 to13 the13 Apps13 page13 13

Page of 40 71 Page of 40 71

13

313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13

13

A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13

13

Page of 42 71 Page of 42 71

The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13

13

The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13

13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

slack

url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5

channel =gt cloud-operations-mon

username =gt nick_cawood

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13

Page of 43 71 Page of 43 71

Page of 44 71 Page of 44 71

In13 Slack13 the13 alerts13 look13 like13 this13

13

Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13

1 Create13 a13 PagerDuty13 integraon13 13

2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13

1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13

13

2 Click13 Add13 New13 Service13

13

313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13

Page of 45 71 Page of 45 71

13

413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13

513 Click13 Add13 Service13

13

The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13

Page of 46 71 Page of 46 71

13

The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt be7eb1ea9c224018923e818d3e891d0b

incident_key =gt logstashhosttype

Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13

Page of 47 71 Page of 47 71

13

Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13

1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13

2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13

3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13

4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13

13 13 http

url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo

http_method =gt post

format =gt json

For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13

13 13 syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

Page of 48 71 Page of 48 71

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13

13

The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13

if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2

This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13

Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13

Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13

Logstash13 severity Shown13 in13 Omnibus13 as

alert Indeterminate

crical Warning

debug 7

emergency Clear

error Minor

informaonal 6

noce Crical

warning Major

Page of 49 71 Page of 49 71

13

Page of 50 71 Page of 50 71

Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13

input

tcp

port =gt 5000

type =gt syslog

filter

mutate

add_field =gt format =gt cf

grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

translate

field =gt data1

destination =gt data1

override =gt true

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

=== Check Alarm Criteria ===

Standard Key Words

if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

=== Alarm Exclusions ===

if [message] =~ (i)sometexthere

mutate remove_tag =gt servicename_alarm

Throttle alarms so we do not hit PD rate limits

Page of 52 71 Page of 52 71

if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 5

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle

after_count =gt 5

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 5

period =gt 1200

key =gt servicename_alarm_warning

add_tag =gt servicename_throttled

Page of 53 71 Page of 53 71

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

output

elasticsearch

hosts =gt localhost

stdout codec =gt rubydebug

if servicename_alarm_err in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

Page of 54 71 Page of 54 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 55 71 Page of 55 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

Page of 56 71 Page of 56 71

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert err message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_error in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

Page of 57 71 Page of 57 71

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 58 71 Page of 58 71

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 59 71 Page of 59 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert error message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

Page of 60 71 Page of 60 71

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

Page of 61 71 Page of 61 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 62 71 Page of 62 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt x

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert fatal message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

Page of 63 71 Page of 63 71

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

Page of 64 71 Page of 64 71

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 65 71 Page of 65 71

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt heavy_exclamation_mark

Page of 66 71 Page of 66 71

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert WARNING message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 67 71 Page of 67 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

Page of 68 71 Page of 68 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 69 71 Page of 69 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

Page of 70 71 Page of 70 71

Page of 71 71 Page of 71 71

Page 11: How to integrate the Elasticsearch Logstash Kibana (ELK ... · How to integrate the Elasticsearch Logstash Kibana (ELK) log analytics stack into IBM Bluemix Nick Cawood IBM Cloud

13

7 Click13 OK13 Then13 click13 CREATE13

The13 virtual13 server13 is13 provisioned13 and13 added13 to13 the13 Bluemix13 dashboard13 under13 VIRTUAL13 SERVERS13

Page of 11 71 Page of 11 71

13

Set13 up13 the13 virtual13 server13

Log13 on13 to13 the13 virtual13 server13 Now13 you13 can13 log13 on13 to13 the13 virtual13 server13 and13 add13 new13 users13 and13 programs13 to13 make13 it13 easier13 to13 access13 the13 virtual13 server13

To13 access13 the13 server13 go13 to13 the13 UNIX13 or13 Linux13 window13 that13 you13 used13 to13 create13 the13 SSH13 key13 In13 the13 directory13 where13 the13 ltuniquekeynamegt13 files13 exist13 run13 the13 following13 command13 referencing13 the13 SSH13 keyname13 the13 public13 IP13 address13 Bluemix13 allocated13 for13 the13 virtual13 server13 and13 the13 password13 for13 the13 SSH13 key13

[userserver ~]$ ssh -i ltuniquekeynamegt ibmcloudltpublicIPaddressgt

Enter13 the13 passphrase13 for13 the13 key13 ltuniquekeynamegt13

Last login Tue Jul 5 125546 2016 from 108-210-96-231lightspeedrlghncsbcglobalnet

[ibmcloudelkforcase ~]$

ibmcloud13 is13 the13 default13 user13 for13 the13 Bluemix13 virtual13 servers13 and13 allows13 SSH13 to13 access13 the13 server13

Acces13 the13 root13 user13 To13 access13 the13 root13 user13 log13 on13 as13 ibmcloud13 user13 and13 run13 the13 following13 command13

[ibmcloudelkforcase ~]$ sudo bash

[rootelkforcase ibmcloud]

The13 root13 user13 can13 be13 used13 to13 add13 programs13 to13 the13 server13 and13 create13 and13 edit13 users13 as13 required13 to13 access13 applicaons13 installed13 on13 the13 virtual13 server13

Page of 12 71 Page of 12 71

Copy13 files13 to13 the13 virtual13 server13 To13 move13 installaon13 files13 to13 the13 server13 go13 to13 the13 UNIX13 or13 Linux13 window13 you13 used13 to13 create13 the13 SSH13 key13 and13 the13 directory13 where13 the13 ltuniquekeynamegt13 files13 exist13 Move13 the13 installaon13 files13 to13 the13 same13 locaon13

Run13 the13 following13 command13 referencing13 the13 SSH13 keyname13 the13 public13 IP13 address13 Bluemix13 allocated13 for13 the13 virtual13 server13 and13 the13 password13 for13 the13 SSH13 key13

[userserver ~]$ scp ndashi ltuniquekeynamegt ltINSTALLATIONFILENAMEgt ibmcloudltpublicIPaddressgthomeibmcloud

Enter13 the13 passphrase13 for13 key13 cloudkey313

This13 copies13 the13 file13 using13 the13 ibmcloud13 user13 to13 the13 homeibmcloud13 directory13 on13 the13 virtual13 server13

Install13 and13 access13 the13 virtual13 server13 via13 VNC13 server13 Using13 the13 ssh13 command13 to13 access13 the13 virtual13 server13 is13 not13 the13 most13 stable13 method13 and13 does13 not13 offer13 X-shy‐Windows13 support13 Therefore13 we13 suggest13 that13 you13 use13 a13 VNC13 client13 to13 access13 the13 virtual13 server13

Installing13 a13 VNC13 client13 should13 be13 done13 as13 the13 root13 user13 Use13 the13 yum13 command13 as13 follows13

[ibmcloudelkforcase ~]$ yum groupinstall GNOME Desktop

[ibmcloudelkforcase ~]$ yum install tigervnc-server

These13 commands13 first13 install13 a13 desktop13 style13 OS13 user13 interface13 and13 then13 a13 VNC13 client13 to13 remotely13 access13 the13 desktop13 style13 OS13

Then13 as13 either13 the13 root13 user13 or13 another13 user13 (example13 below13 uses13 elk)13 run13 the13 following13 command13 to13 create13 the13 VNC13 client13 connecon13

[elkelkforcase ~]$ vncserver

New elkforcase2 (elk) desktop is elkforcase2

Starting applications specified in homeelkvncxstartup

Log file is homeelkvncelkforcase2log

[elkelkforcase ~]$

Install13 a13 VNC13 client13 on13 the13 client13 machine13 used13 to13 access13 the13 Internet13 and13 (in13 the13 example13 above)13 use13 the13 command13 ltpublicIPadddressgt213 and13 the13 password13 for13 ELK13 to13 access13 the13 virtual13 server13

Using13 a13 VNC13 client13 means13 that13 you13 can13 leave13 long13 duraon13 installaons13 running13 without13 fear13 of13 the13 terminal13 losing13 connecon13 and13 the13 installaons13 failing13

Install13 the13 ELK13 stack13 Download13 the13 source13 soaware13 code13 for13 the13 ELK13 stack13 from13 the13 ELK13 website13 13

At13 the13 me13 of13 wring13 the13 downloaded13 soaware13 files13 were13

Page of 13 71 Page of 13 71

13

Transfer13 the13 files13 to13 the13 virtual13 server13 (using13 the13 scp13 method13 detailed13 above)13

Install13 Elas0csearch13 As13 a13 root13 user13 create13 a13 non-shy‐root13 user13 with13 the13 adduser13 command13

Create13 a13 directory13 to13 install13 Elascsearch13 into13 and13 untar13 the13 contents13 of13 the13 Elascsearch13 install13 file13 into13 that13 directory13

[elkelkforcase]$ cd home

[elkelkforcase]$ mkdir elasticsearch

[elkelkforcase]$ cp homeibmcloudelasticsearch-233targz homeelasticsearch

[elkelkforcase]$ cd elasticsearch

[elkelkforcase]$ gunzip elasticsearch-233targz

[elkelkforcase]$ untar elasticsearch-233tar

Elascsearch13 is13 now13 installed13

Start13 Elas0csearch13 To13 start13 Elascsearch13 go13 to13 the13 bin13 directory13 as13 the13 non-shy‐root13 user13 when13 you13 installed13 it13 and13 run13 the13 elascsearch13 executable13

[elkelkforcase]$ cd homeelasticsearchelasticsearch-233bin

[elkelkforcase]$ elasticsearch amp

[2] 22211

[elkelkforcase]$ [2016-07-05 133857730][INFO ][node] [Arsenal] version[233] pid[22211] build[218bdf12016-05-17T154004Z]

[2016-07-05 133857740][INFO ][node] [Arsenal] initializing

[2016-07-05 133858984][INFO ][plugins] [Arsenal] modules [reindex lang-expression lang-groovy] plugins [] sites []

[2016-07-05 133859146][INFO ][env] [Arsenal] using [1] data paths mounts [[ (rootfs)]] net usable_space [155gb] net total_space [199gb] spins [unknown] types [rootfs]

[2016-07-05 133859146][INFO ][env] [Arsenal] heap size [10156mb] compressed ordinary object pointers [true]

Page of 14 71 Page of 14 71

[2016-07-05 133859147][WARN ][env] [Arsenal] max file descriptors [4096] for elasticsearch process likely too low consider increasing to at least [65536]

[2016-07-05 133903526][INFO ][node] [Arsenal] initialized

[2016-07-05 133903526][INFO ][node] [Arsenal] starting

To13 test13 that13 Elascsearch13 is13 running13 correctly13 open13 the13 browser13 in13 the13 VNC13 client13 and13 run13 localhost920013

13

Install13 Kibana13 ELK13 recommends13 installing13 and13 running13 Kibana13 as13 root13 13

1 As13 root13 user13 create13 an13 installaon13 directory13 and13 untar13 the13 installaon13 file13

[rootelkforcase]$ cd home

[rootelkforcase]$ mkdir kibana

[rootelkforcase]$ cp homeibmcloudkibana-451-linux-x64targz homekibana

[rootelkforcase]$ cd kibana

[rootelkforcase]$ gunzip kibana-451-linux-x64targz

[rootelkforcase]$ untar kibana-451-linux-x64tar

2 Modify13 the13 manifestyml13 configuraon13 file13 Go13 to13 the13 kibana-shy‐451-shy‐linux-shy‐x6413 directory13

[rootelkforcase]$ cd homekibanakibana-451-linux-x64

3 Edit13 the13 manifestyml13 file13 so13 the13 contents13 are13

applicaons13

-shy‐ name13 kibana-shy‐in-shy‐bluemix13

memory13 128M13

host13 kibana-shy‐in-shy‐bluemix13

buildpack13 hcpsgithubcomcloudfoundry-shy‐communitynginx-shy‐buildpackgit13

Page of 15 71 Page of 15 71

4 Edit13 the13 configkibanayml13 file13 so13 the13 contents13 are13

Kibana is served by a back end server This controls which port to use

port 5601

The host to bind the server to

host 0000

If you are running kibana behind a proxy and want to mount it at a path

specify that path here The basePath cant end in a slash

serverbasePath

The maximum payload size in bytes on incoming server requests

servermaxPayloadBytes 1048576

The Elasticsearch instance to use for all your queries

elasticsearchurl httplocalhost9200

preserve_elasticsearch_host true will send the hostname specified in `elasticsearch` If you set it to false

then the host you use to connect to this Kibana instance will be sent

elasticsearchpreserveHost true

Kibana uses an index in Elasticsearch to store saved searches visualizations

and dashboards It will create a new index if it doesnt already exist

kibanaindex kibana

The default application to load

kibanadefaultAppId discover

If your Elasticsearch is protected with basic auth these are the user credentials

used by the Kibana server to perform maintenance on the kibana_index at startup Your

Kibana users will still need to authenticate with Elasticsearch (which is proxied through

the Kibana server)

Page of 16 71 Page of 16 71

elasticsearchusername user

elasticsearchpassword pass

SSL for outgoing requests from the Kibana Server to the browser (PEM formatted)

serversslcert pathtoyourservercrt

serversslkey pathtoyourserverkey

Optional setting to validate that your Elasticsearch backend uses the same key files (PEM formatted)

elasticsearchsslcert pathtoyourclientcrt

elasticsearchsslkey pathtoyourclientkey

If you need to provide a CA certificate for your Elasticsearch instance put

the path of the pem file here

elasticsearchsslca pathtoyourCApem

Set to false to have a complete disregard for the validity of the SSL

certificate

elasticsearchsslverify true

Time in milliseconds to wait for elasticsearch to respond to pings defaults to

request_timeout setting

elasticsearchpingTimeout 1500

Time in milliseconds to wait for responses from the back end or elasticsearch

This must be gt 0

elasticsearchrequestTimeout 30000

Time in milliseconds for Elasticsearch to wait for responses from shards

Set to 0 to disable

elasticsearchshardTimeout 0

Time in milliseconds to wait for Elasticsearch at Kibana startup before retrying

elasticsearchstartupTimeout 5000

elasticsearchstartupTimeout 30000

Page of 17 71 Page of 17 71

Set the path to where you would like the process id file to be created

pidfile varrunkibanapid

If you would like to send the log output to a file you can set the path below

loggingdest stdout

Set this to true to suppress all logging output

loggingsilent false

Set this to true to suppress all logging output except for error messages

loggingquiet false

Set this to true to log all events including system usage information and all requests

loggingverbose false

Kibana13 is13 now13 installed13

Start13 Kibana13 To13 start13 Kibana13 go13 to13 the13 bin13 directory13 and13 run13 the13 kibana13 executable13

[rootelkforcase]$

[rootelkforcase]$ cd homekibanakibana-451-linux-x64bin

[rootelkforcase]$ kibana amp

[1] 22283

[rootelkforcase] log [095900542] [info][status][pluginkibana] Status changed from uninitialized to green - Ready

log [095900600] [info][status][pluginelasticsearch] Status changed from uninitialized to yellow - Waiting for Elasticsearch

log [095900689] [info][status][pluginkbn_vislib_vis_types] Status changed from uninitialized to green - Ready

log [095900717] [info][status][pluginmarkdown_vis] Status changed from uninitialized to green - Ready

log [095900721] [info][status][pluginmetric_vis] Status changed from uninitialized to green - Ready

log [095900814] [info][status][pluginspyModes] Status changed from uninitialized to green - Ready

log [095900934] [info][status][pluginstatusPage] Status changed from uninitialized to green - Ready

Page of 18 71 Page of 18 71

log [095900941] [info][status][plugintable_vis] Status changed from uninitialized to green - Ready

log [095901067] [info][listening] Server running at http00005601

log [095906083] [info][status][pluginelasticsearch] Status changed from yellow to yellow - No existing Kibana index found

log [095908942] [info][status][pluginelasticsearch] Status changed from yellow to green - Kibana index ready

Kibana13 is13 now13 started13

Broadcast13 Bluemix13 app13 log13 records13 Bluemix13 apps13 (which13 are13 shown13 in13 the13 US13 South13 Bluemix13 region13 of13 the13 System13 Architecture13 diagram)13 will13 produce13 logs13 in13 a13 syslog13 rfc542413 format13 using13 a13 Bluemix13 tool13 called13 Loggregator13

For13 Logstash13 to13 receive13 these13 logs13 each13 Bluemix13 app13 needs13 to13 broadcast13 these13 logs13 To13 do13 that13 run13 the13 following13 commands13 on13 each13 Bluemix13 app13 using13 the13 public13 IP13 address13 of13 the13 ELK13 virtual13 server13 the13 ELK13 tag13 to13 trace13 the13 process13 and13 insert13 the13 name13 of13 each13 app13 in13 turn13

cf cups elk -l syslogltpublicIPaddress5000 cf bind-service ltmyappgt elk

cf restage ltmyappgt

Install13 Logstash13 To13 install13 Logstash13 as13 root13 user13 create13 a13 directory13 for13 the13 installaon13 and13 untar13 the13 installaon13 file13 there13

13 13 [rootelkforcase]$ cd home

[rootelkforcase]$ mkdir logstash

[rootelkforcase]$ cp homeibmcloudlogstash-233targz homelogstash

[rootelkforcase]$ cd logstash

[rootelkforcase]$ gunzip logstash-233targz

[rootelkforcase]$ untar logstash-233tar

Write13 Logstash13 configura0on13 Logstash13 requires13 a13 configuraon13 file13 to13 be13 wricen13 to13 receive13 transform13 and13 output13 the13 log13 records13 to13 Elascsearch13 This13 secon13 explains13 the13 basic13 requirements13 of13 processing13 Bluemix13 app13 log13 records13 into13 Elascsearch13 (more13 complex13 configuraons13 are13 detailed13 later13 in13 this13 document13 for13 the13 integraons13 with13 third-shy‐party13 tools)13

Create13 a13 conf13 file13 as13 root13 user13

[rootelkforcase]$ cd homelogstashlogstash-233

[rootelkforcase]$ mkdir config

Page of 19 71 Page of 19 71

[rootelkforcase]$ cd config

[rootelkforcase]$ touch loggregatorconf

Edit13 the13 loggregator13 file13 with13 the13 following13 contents13

13 Input13 secon13 defines13 which13 log13 records13 to13 receive13

13 port13 and13 type13 reflect13 method13 selected13 for13 Cloud13 Foundry13 to13 broadcast13 the13 records13

input

tcp

port =gt 5000

type =gt syslog

13 Filter13 secon13 ndash13 transforms13 the13 data13

filter

Add a field to identify the records as Cloud Foundry based

mutate

add_field =gt format =gt cf

13 Output13 secon13 ndash13 sends13 the13 data13 to13 its13 desnaon(s)13

output

13 Use13 the13 elascsearch13 plugin13 to13 send13 the13 data13 to13 the13 elascsearch13 instance13

13 elasticsearch

hosts =gt localhost

stdout codec =gt rubydebug

13 End13 of13 Config13

Save13 the13 file13 Verify13 the13 format13 of13 the13 file13 is13 acceptable13 by13 running13 the13 following13 commands13

[rootelkforcase]$ cd homelogstashlogstash-233

[rootelkforcase]$ binlogstash agent --configtest -f configloggregatorconf

Page of 20 71 Page of 20 71

Configuraon13 is13 OK13

Start13 Logstash13 Aaer13 you13 have13 wricen13 and13 verified13 the13 Logstash13 configuraon13 you13 can13 start13 Logstash13 to13 process13 records13 from13 the13 Bluemix13 apps13 (via13 Cloud13 Foundry)13 Run13 the13 following13 as13 root13 user13

[rootelkforcase]$ cd homelogstashlogstash-233

[rootelkforcase]$ binlogstash agent -f configloggregatorconf

Logstash13 is13 now13 started13

Monitor13 Logstash13 processing13 Once13 Logstash13 is13 running13 and13 log13 records13 are13 being13 produced13 by13 the13 Bluemix13 apps13 you13 can13 monitor13 progress13 in13 the13 same13 window13 that13 was13 used13 to13 start13 Logstash13 Output13 is13 similar13 to13 the13 following13

13

The13 log13 records13 being13 broadcast13 from13 the13 Bluemix13 apps13 are13 now13 sent13 to13 Elascsearch13 from13 the13 Logstash13 output13 configuraon13 Records13 are13 sent13 in13 real13 me13 If13 Logstash13 is13 not13 running13 while13 log13 records13 are13 being13 produced13 then13 these13 records13 will13 not13 be13 processed13 by13 Logstash13 and13 will13 not13 be13 stored13 in13 Elascsearch13

Page of 21 71 Page of 21 71

Use13 ELK13 The13 log13 records13 from13 the13 Bluemix13 apps13 are13 now13 being13 stored13 and13 indexed13 by13 the13 Elascsearch13 component13 This13 data13 can13 be13 accessed13 via13 the13 Kibana13 applicaon13

Kibana13 Discover13 page13 Using13 a13 web13 browser13 either13 directly13 from13 a13 client13 or13 using13 the13 VNC13 client13 access13 Kibana13 using13 the13 publicIPaddress13 of13 the13 ELK13 virtual13 server13

hcpltpublicIPaddressgt560113

No13 userpassword13 credenals13 are13 required13

Kibana13 defaults13 to13 a13 search13 of13 all13 records13 ()13 in13 the13 last13 1513 minutes13 and13 returns13 the13 results13

13

More13 complex13 searches13 are13 possible13 by13 changing13 the13 me13 filter13 and13 supplying13 search13 queries13

Change13 the13 0me13 filter13 To13 change13 the13 me13 ranges13 click13 the13 clock13 symbol13 in13 the13 top-shy‐right13 of13 the13 Kibana13 Discover13 page13

13

You13 will13 see13 the13 following13 opons13

13

Page of 22 71 Page of 22 71

A13 variety13 of13 quick13 me13 ranges13 can13 be13 selected13 or13 you13 can13 switch13 to13 relave13 or13 absolute13 me13 ranges13

Quick13 me13 ranges13 allow13 switching13 between13 last13 X13 minutes13 to13 last13 Y13 hours13 or13 Z13 days13 etc13

Relave13 me13 ranges13 allow13 more13 specific13 versions13 of13 the13 Quick13 opons13 from13 specific13 minuteshoursdays13 to13 now13

13

Absolute13 me13 ranges13 specify13 the13 start13 and13 end13 mes13 for13 the13 searches13

13

Custom13 search13 queries13 You13 can13 use13 the13 text13 window13 of13 the13 Discover13 page13 to13 search13 for13 matches13 within13 the13 log13 records13 You13 can13 use13 Apache13 Lucene13 or13 Elascsearch13 query13 DSL13 syntax13

13

Page of 23 71 Page of 23 71

Saved13 searches13 You13 can13 save13 any13 custom13 search13 syntax13 and13 choice13 of13 me13 range13 for13 future13 ease13 of13 use13 Any13 relave13 or13 absolute13 me13 frame13 is13 saved13 as13 part13 of13 the13 search13 13

To13 save13 a13 search13

1 Set13 up13 and13 run13 the13 required13 syntax13 and13 me13 frame13

13

2 Click13 the13 Save13 Search13 icon13

13

3 Type13 in13 a13 name13 for13 the13 saved13 search13

Page of 24 71 Page of 24 71

13

7 To13 delete13 or13 update13 saved13 searches13 click13 13 manage13 saved13 searches13

13

Use13 the13 edit13 icons13 to13 make13 any13 changes13

Page of 26 71 Page of 26 71

13

Build13 simple13 Kibana13 histograms13 Kibana13 allows13 you13 to13 build13 charts13 and13 dashboards13 based13 on13 the13 search13 capability13 offered13 by13 the13 Discover13 page13 Saved13 searches13 can13 be13 used13 as13 the13 basis13 for13 charts13 and13 dashboards13 built13 using13 those13 charts13 To13 build13 a13 simple13 histogram13

1 Select13 the13 Visualize13 page13

13

2 Select13 Ver0cal13 Bar13 Chart13

13

3 To13 use13 a13 saved13 search13 click13 From13 a13 saved13 search13

Page of 27 71 Page of 27 71

13

4 Select13 a13 previously13 saved13 search13

13

5 Select13 to13 edit13 the13 x-shy‐axis

6 Edit13 the13 fields13 with13 informaon13 about13 the13 aggregaon13 field13 interval13 and13 custom13 label13

Page of 28 71 Page of 28 71

13

Page of 29 71 Page of 29 71

7 Click13 the13 green13 arrow13 The13 histogram13 is13 created13 and13 run13 based13 on13 the13 saved13 search

13

8 To13 save13 the13 histogram13 click13 the13 Save13 Visualiza0on13 icon13 and13 add13 a13 tle13

13

13

9 Click13 Save13

10 To13 view13 the13 visualisaon13 click13 the13 Dashboard13 page13

Page of 30 71 Page of 30 71

13

11 Click13 the13 Add13 Visualiza0on13 icon13

13

12 Select13 the13 visiualizaon13 that13 you13 just13 created13

13

Page of 31 71 Page of 31 71

13

13 To13 save13 the13 dashboard13 click13 the13 Save13 Dashboard13 icon13

Page of 32 71 Page of 32 71

13

14 Add13 a13 tle13 and13 select13 whether13 to13 store13 the13 selected13 me13 frame13 with13 the13 dashboard13

13

15 Click13 Save13

16 You13 can13 edit13 the13 shape13 and13 number13 of13 charts13 and13 save13 the13 dashboard13

17 To13 re-shy‐run13 the13 dashboard13 select13 the13 Dashboard13 page13 and13 click13 the13 Load13 Saved13 Dashboard13 icon13

Page of 33 71 Page of 33 71

13

18 Select13 the13 saved13 dashboard13 from13 the13 list13

13

19 The13 dashboard13 runs13

13

Page of 34 71 Page of 34 71

Page of 35 71 Page of 35 71

Aler0ng13 with13 ELK13 ELK13 examines13 the13 log13 records13 in13 process13 to13 send13 alerts13 to13 third-shy‐party13 alerng13 event13 management13 and13 social13 media13 applicaons13 As13 shown13 previously13 in13 the13 ELK13 architecture13 the13 ELK13 strategy13 for13 alerng13 is13 to13 send13 alerts13 from13 the13 Logstash13 component13 directly13 Therefore13 log13 records13 and13 alerts13 are13 sent13 to13 both13 Elascsearch13 and13 third-shy‐party13 tools13 at13 the13 same13 instant13 using13 the13 output13 secon13 of13 the13 Logstash13 configuraon13 Any13 alerts13 are13 generated13 during13 the13 filter13 secon13 processing13 of13 the13 Logstash13 configuraon13

Create13 alerts13 in13 Logstash13 The13 Logstash13 filter13 secon13 is13 used13 to13 trigger13 alerts13 based13 on13 the13 log13 records13 being13 processed13 For13 example13 log13 records13 can13 be13 checked13 for13 errors13 warnings13 crashed13 or13 stopped13 messages13 and13 generate13 an13 alert13

Here13 is13 an13 example13 Logstash13 filter13 code13

13 if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

The13 specific13 tags13 are13 set13 against13 the13 log13 records13 to13 trigger13 the13 alert13 later13 in13 the13 configuraon13 in13 the13 output13 secon13

Control13 the13 number13 of13 generated13 alerts13 using13 the13 Logstash13 throcle13 plugin13 so13 that13 either13 a13 lot13 of13 instances13 in13 a13 fixed13 me13 period13 are13 required13 before13 an13 alert13 is13 sent13 or13 a13 specific13 number13 of13 instances13 triggers13 the13 alert13

Page of 36 71 Page of 36 71

Edit13 the13 plugin13 parameters13 to13 meet13 system13 requirements13

13 13 13 13 13 if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 10

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle13

after_count =gt 20

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 30

period =gt 1200

key =gt servicename_alarm_warning

Page of 37 71 Page of 37 71

add_tag =gt servicename_throttled

13 13 13 13 13 13 13 13 13

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 40

period =gt 1500

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

Enrich13 alerts13 with13 message13 detail13 Some13 of13 the13 third-shy‐party13 integraons13 benefit13 from13 having13 components13 of13 the13 raw13 log13 record13 sent13 in13 addion13 to13 the13 raw13 record13 or13 fields13 in13 the13 record13 edited13 transformed13 or13 translated13

Grok13 Logstash13 allows13 the13 log13 record13 to13 be13 broken13 up13 into13 logical13 secons13 and13 each13 secon13 stored13 in13 a13 new13 parameter13 using13 the13 grok13 parameter13 in13 the13 filter13 secon13 Here13 is13 an13 example13 grok13

13 grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

To13 build13 the13 grok13 statement13 Logstash13 offers13 an13 online13 tool13 at13 hcpsgrokdebugherokuappcom13

Translate13 If13 specific13 fields13 in13 the13 log13 record13 are13 coded13 (for13 example13 the13 Bluemix13 app13 name13 is13 oaen13 not13 sent13 in13 the13 syslog13 messages13 instead13 the13 GUID13 is13 sent13 which13 is13 not13 useful)13 then13 the13 filter13 secon13 translate13 plugin13 can13 be13 used13 to13 add13 new13 fields13 by13 holding13 the13 decoded13 versions13 of13 those13 values13

13 translate

field =gt data1

destination =gt data1

override =gt true

Page of 38 71 Page of 38 71

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

dc50f6e4-c052-436a-bf27-95ba392c4228 monverifyservicemanagement

0c677e82-5bfc-4e38-9155-e416249e3ee6 ms-catalog-postpyretic-pomiculture

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

However13 the13 translate13 plugin13 is13 not13 included13 with13 the13 Logstash13 installaon13 In13 the13 next13 secon13 I13 will13 13 describe13 how13 to13 add13 new13 plugins13 to13 the13 Logstash13 instance13

Add13 custom13 plugins13 to13 Logstash13 As13 an13 open13 source13 tool13 Logstash13 encourages13 users13 to13 develop13 content13 for13 all13 users13 to13 share13 Some13 of13 the13 plugins13 that13 were13 created13 by13 our13 community13 include13 a13 translaon13 plugin13 an13 output13 Slack13 plugin13 and13 an13 output13 syslog13 plugin13 Letrsquos13 look13 at13 how13 to13 add13 these13 plugins13 to13 your13 environment13

Filter13 translate13 plugin13 To13 add13 the13 translate13 plugin13

1 Download13 the13 plugin13

2 Copy13 the13 plugin13 file13 logstash-shy‐filter-shy‐translate-shy‐214gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-filter-translate 214

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Output13 Slack13 plugin13 To13 add13 the13 Slack13 plugin13

Page of 39 71 Page of 39 71

1 Download13 the13 plugin13 13

2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐slack-shy‐014gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-output-slack 014

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Output13 syslog13 plugin13 To13 add13 the13 syslog13 plugin13

1 Download13 the13 plugin13 13

2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐syslog-shy‐214gem13 to13 the13 ELK13 Virtual13 Server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-output-syslog 214

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Integrate13 Logstash13 with13 Slack13 To13 send13 generated13 alerts13 to13 Slack13 complete13 these13 steps13

1 Create13 a13 Slack13 integraon13 13

2 Use13 the13 Slack13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 Slack13 integraon13

113 Login13 to13 Slack13 and13 go13 to13 the13 Apps13 page13 13

Page of 40 71 Page of 40 71

13

313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13

13

A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13

13

Page of 42 71 Page of 42 71

The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13

13

The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13

13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

slack

url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5

channel =gt cloud-operations-mon

username =gt nick_cawood

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13

Page of 43 71 Page of 43 71

Page of 44 71 Page of 44 71

In13 Slack13 the13 alerts13 look13 like13 this13

13

Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13

1 Create13 a13 PagerDuty13 integraon13 13

2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13

1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13

13

2 Click13 Add13 New13 Service13

13

313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13

Page of 45 71 Page of 45 71

13

413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13

513 Click13 Add13 Service13

13

The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13

Page of 46 71 Page of 46 71

13

The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt be7eb1ea9c224018923e818d3e891d0b

incident_key =gt logstashhosttype

Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13

Page of 47 71 Page of 47 71

13

Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13

1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13

2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13

3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13

4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13

13 13 http

url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo

http_method =gt post

format =gt json

For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13

13 13 syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

Page of 48 71 Page of 48 71

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13

13

The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13

if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2

This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13

Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13

Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13

Logstash13 severity Shown13 in13 Omnibus13 as

alert Indeterminate

crical Warning

debug 7

emergency Clear

error Minor

informaonal 6

noce Crical

warning Major

Page of 49 71 Page of 49 71

13

Page of 50 71 Page of 50 71

Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13

input

tcp

port =gt 5000

type =gt syslog

filter

mutate

add_field =gt format =gt cf

grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

translate

field =gt data1

destination =gt data1

override =gt true

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

=== Check Alarm Criteria ===

Standard Key Words

if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

=== Alarm Exclusions ===

if [message] =~ (i)sometexthere

mutate remove_tag =gt servicename_alarm

Throttle alarms so we do not hit PD rate limits

Page of 52 71 Page of 52 71

if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 5

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle

after_count =gt 5

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 5

period =gt 1200

key =gt servicename_alarm_warning

add_tag =gt servicename_throttled

Page of 53 71 Page of 53 71

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

output

elasticsearch

hosts =gt localhost

stdout codec =gt rubydebug

if servicename_alarm_err in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

Page of 54 71 Page of 54 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 55 71 Page of 55 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

Page of 56 71 Page of 56 71

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert err message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_error in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

Page of 57 71 Page of 57 71

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 58 71 Page of 58 71

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 59 71 Page of 59 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert error message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

Page of 60 71 Page of 60 71

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

Page of 61 71 Page of 61 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 62 71 Page of 62 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt x

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert fatal message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

Page of 63 71 Page of 63 71

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

Page of 64 71 Page of 64 71

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 65 71 Page of 65 71

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt heavy_exclamation_mark

Page of 66 71 Page of 66 71

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert WARNING message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 67 71 Page of 67 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

Page of 68 71 Page of 68 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 69 71 Page of 69 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

Page of 70 71 Page of 70 71

Page of 71 71 Page of 71 71

Page 12: How to integrate the Elasticsearch Logstash Kibana (ELK ... · How to integrate the Elasticsearch Logstash Kibana (ELK) log analytics stack into IBM Bluemix Nick Cawood IBM Cloud

13

Set13 up13 the13 virtual13 server13

Log13 on13 to13 the13 virtual13 server13 Now13 you13 can13 log13 on13 to13 the13 virtual13 server13 and13 add13 new13 users13 and13 programs13 to13 make13 it13 easier13 to13 access13 the13 virtual13 server13

To13 access13 the13 server13 go13 to13 the13 UNIX13 or13 Linux13 window13 that13 you13 used13 to13 create13 the13 SSH13 key13 In13 the13 directory13 where13 the13 ltuniquekeynamegt13 files13 exist13 run13 the13 following13 command13 referencing13 the13 SSH13 keyname13 the13 public13 IP13 address13 Bluemix13 allocated13 for13 the13 virtual13 server13 and13 the13 password13 for13 the13 SSH13 key13

[userserver ~]$ ssh -i ltuniquekeynamegt ibmcloudltpublicIPaddressgt

Enter13 the13 passphrase13 for13 the13 key13 ltuniquekeynamegt13

Last login Tue Jul 5 125546 2016 from 108-210-96-231lightspeedrlghncsbcglobalnet

[ibmcloudelkforcase ~]$

ibmcloud13 is13 the13 default13 user13 for13 the13 Bluemix13 virtual13 servers13 and13 allows13 SSH13 to13 access13 the13 server13

Acces13 the13 root13 user13 To13 access13 the13 root13 user13 log13 on13 as13 ibmcloud13 user13 and13 run13 the13 following13 command13

[ibmcloudelkforcase ~]$ sudo bash

[rootelkforcase ibmcloud]

The13 root13 user13 can13 be13 used13 to13 add13 programs13 to13 the13 server13 and13 create13 and13 edit13 users13 as13 required13 to13 access13 applicaons13 installed13 on13 the13 virtual13 server13

Page of 12 71 Page of 12 71

Copy13 files13 to13 the13 virtual13 server13 To13 move13 installaon13 files13 to13 the13 server13 go13 to13 the13 UNIX13 or13 Linux13 window13 you13 used13 to13 create13 the13 SSH13 key13 and13 the13 directory13 where13 the13 ltuniquekeynamegt13 files13 exist13 Move13 the13 installaon13 files13 to13 the13 same13 locaon13

Run13 the13 following13 command13 referencing13 the13 SSH13 keyname13 the13 public13 IP13 address13 Bluemix13 allocated13 for13 the13 virtual13 server13 and13 the13 password13 for13 the13 SSH13 key13

[userserver ~]$ scp ndashi ltuniquekeynamegt ltINSTALLATIONFILENAMEgt ibmcloudltpublicIPaddressgthomeibmcloud

Enter13 the13 passphrase13 for13 key13 cloudkey313

This13 copies13 the13 file13 using13 the13 ibmcloud13 user13 to13 the13 homeibmcloud13 directory13 on13 the13 virtual13 server13

Install13 and13 access13 the13 virtual13 server13 via13 VNC13 server13 Using13 the13 ssh13 command13 to13 access13 the13 virtual13 server13 is13 not13 the13 most13 stable13 method13 and13 does13 not13 offer13 X-shy‐Windows13 support13 Therefore13 we13 suggest13 that13 you13 use13 a13 VNC13 client13 to13 access13 the13 virtual13 server13

Installing13 a13 VNC13 client13 should13 be13 done13 as13 the13 root13 user13 Use13 the13 yum13 command13 as13 follows13

[ibmcloudelkforcase ~]$ yum groupinstall GNOME Desktop

[ibmcloudelkforcase ~]$ yum install tigervnc-server

These13 commands13 first13 install13 a13 desktop13 style13 OS13 user13 interface13 and13 then13 a13 VNC13 client13 to13 remotely13 access13 the13 desktop13 style13 OS13

Then13 as13 either13 the13 root13 user13 or13 another13 user13 (example13 below13 uses13 elk)13 run13 the13 following13 command13 to13 create13 the13 VNC13 client13 connecon13

[elkelkforcase ~]$ vncserver

New elkforcase2 (elk) desktop is elkforcase2

Starting applications specified in homeelkvncxstartup

Log file is homeelkvncelkforcase2log

[elkelkforcase ~]$

Install13 a13 VNC13 client13 on13 the13 client13 machine13 used13 to13 access13 the13 Internet13 and13 (in13 the13 example13 above)13 use13 the13 command13 ltpublicIPadddressgt213 and13 the13 password13 for13 ELK13 to13 access13 the13 virtual13 server13

Using13 a13 VNC13 client13 means13 that13 you13 can13 leave13 long13 duraon13 installaons13 running13 without13 fear13 of13 the13 terminal13 losing13 connecon13 and13 the13 installaons13 failing13

Install13 the13 ELK13 stack13 Download13 the13 source13 soaware13 code13 for13 the13 ELK13 stack13 from13 the13 ELK13 website13 13

At13 the13 me13 of13 wring13 the13 downloaded13 soaware13 files13 were13

Page of 13 71 Page of 13 71

13

Transfer13 the13 files13 to13 the13 virtual13 server13 (using13 the13 scp13 method13 detailed13 above)13

Install13 Elas0csearch13 As13 a13 root13 user13 create13 a13 non-shy‐root13 user13 with13 the13 adduser13 command13

Create13 a13 directory13 to13 install13 Elascsearch13 into13 and13 untar13 the13 contents13 of13 the13 Elascsearch13 install13 file13 into13 that13 directory13

[elkelkforcase]$ cd home

[elkelkforcase]$ mkdir elasticsearch

[elkelkforcase]$ cp homeibmcloudelasticsearch-233targz homeelasticsearch

[elkelkforcase]$ cd elasticsearch

[elkelkforcase]$ gunzip elasticsearch-233targz

[elkelkforcase]$ untar elasticsearch-233tar

Elascsearch13 is13 now13 installed13

Start13 Elas0csearch13 To13 start13 Elascsearch13 go13 to13 the13 bin13 directory13 as13 the13 non-shy‐root13 user13 when13 you13 installed13 it13 and13 run13 the13 elascsearch13 executable13

[elkelkforcase]$ cd homeelasticsearchelasticsearch-233bin

[elkelkforcase]$ elasticsearch amp

[2] 22211

[elkelkforcase]$ [2016-07-05 133857730][INFO ][node] [Arsenal] version[233] pid[22211] build[218bdf12016-05-17T154004Z]

[2016-07-05 133857740][INFO ][node] [Arsenal] initializing

[2016-07-05 133858984][INFO ][plugins] [Arsenal] modules [reindex lang-expression lang-groovy] plugins [] sites []

[2016-07-05 133859146][INFO ][env] [Arsenal] using [1] data paths mounts [[ (rootfs)]] net usable_space [155gb] net total_space [199gb] spins [unknown] types [rootfs]

[2016-07-05 133859146][INFO ][env] [Arsenal] heap size [10156mb] compressed ordinary object pointers [true]

Page of 14 71 Page of 14 71

[2016-07-05 133859147][WARN ][env] [Arsenal] max file descriptors [4096] for elasticsearch process likely too low consider increasing to at least [65536]

[2016-07-05 133903526][INFO ][node] [Arsenal] initialized

[2016-07-05 133903526][INFO ][node] [Arsenal] starting

To13 test13 that13 Elascsearch13 is13 running13 correctly13 open13 the13 browser13 in13 the13 VNC13 client13 and13 run13 localhost920013

13

Install13 Kibana13 ELK13 recommends13 installing13 and13 running13 Kibana13 as13 root13 13

1 As13 root13 user13 create13 an13 installaon13 directory13 and13 untar13 the13 installaon13 file13

[rootelkforcase]$ cd home

[rootelkforcase]$ mkdir kibana

[rootelkforcase]$ cp homeibmcloudkibana-451-linux-x64targz homekibana

[rootelkforcase]$ cd kibana

[rootelkforcase]$ gunzip kibana-451-linux-x64targz

[rootelkforcase]$ untar kibana-451-linux-x64tar

2 Modify13 the13 manifestyml13 configuraon13 file13 Go13 to13 the13 kibana-shy‐451-shy‐linux-shy‐x6413 directory13

[rootelkforcase]$ cd homekibanakibana-451-linux-x64

3 Edit13 the13 manifestyml13 file13 so13 the13 contents13 are13

applicaons13

-shy‐ name13 kibana-shy‐in-shy‐bluemix13

memory13 128M13

host13 kibana-shy‐in-shy‐bluemix13

buildpack13 hcpsgithubcomcloudfoundry-shy‐communitynginx-shy‐buildpackgit13

Page of 15 71 Page of 15 71

4 Edit13 the13 configkibanayml13 file13 so13 the13 contents13 are13

Kibana is served by a back end server This controls which port to use

port 5601

The host to bind the server to

host 0000

If you are running kibana behind a proxy and want to mount it at a path

specify that path here The basePath cant end in a slash

serverbasePath

The maximum payload size in bytes on incoming server requests

servermaxPayloadBytes 1048576

The Elasticsearch instance to use for all your queries

elasticsearchurl httplocalhost9200

preserve_elasticsearch_host true will send the hostname specified in `elasticsearch` If you set it to false

then the host you use to connect to this Kibana instance will be sent

elasticsearchpreserveHost true

Kibana uses an index in Elasticsearch to store saved searches visualizations

and dashboards It will create a new index if it doesnt already exist

kibanaindex kibana

The default application to load

kibanadefaultAppId discover

If your Elasticsearch is protected with basic auth these are the user credentials

used by the Kibana server to perform maintenance on the kibana_index at startup Your

Kibana users will still need to authenticate with Elasticsearch (which is proxied through

the Kibana server)

Page of 16 71 Page of 16 71

elasticsearchusername user

elasticsearchpassword pass

SSL for outgoing requests from the Kibana Server to the browser (PEM formatted)

serversslcert pathtoyourservercrt

serversslkey pathtoyourserverkey

Optional setting to validate that your Elasticsearch backend uses the same key files (PEM formatted)

elasticsearchsslcert pathtoyourclientcrt

elasticsearchsslkey pathtoyourclientkey

If you need to provide a CA certificate for your Elasticsearch instance put

the path of the pem file here

elasticsearchsslca pathtoyourCApem

Set to false to have a complete disregard for the validity of the SSL

certificate

elasticsearchsslverify true

Time in milliseconds to wait for elasticsearch to respond to pings defaults to

request_timeout setting

elasticsearchpingTimeout 1500

Time in milliseconds to wait for responses from the back end or elasticsearch

This must be gt 0

elasticsearchrequestTimeout 30000

Time in milliseconds for Elasticsearch to wait for responses from shards

Set to 0 to disable

elasticsearchshardTimeout 0

Time in milliseconds to wait for Elasticsearch at Kibana startup before retrying

elasticsearchstartupTimeout 5000

elasticsearchstartupTimeout 30000

Page of 17 71 Page of 17 71

Set the path to where you would like the process id file to be created

pidfile varrunkibanapid

If you would like to send the log output to a file you can set the path below

loggingdest stdout

Set this to true to suppress all logging output

loggingsilent false

Set this to true to suppress all logging output except for error messages

loggingquiet false

Set this to true to log all events including system usage information and all requests

loggingverbose false

Kibana13 is13 now13 installed13

Start13 Kibana13 To13 start13 Kibana13 go13 to13 the13 bin13 directory13 and13 run13 the13 kibana13 executable13

[rootelkforcase]$

[rootelkforcase]$ cd homekibanakibana-451-linux-x64bin

[rootelkforcase]$ kibana amp

[1] 22283

[rootelkforcase] log [095900542] [info][status][pluginkibana] Status changed from uninitialized to green - Ready

log [095900600] [info][status][pluginelasticsearch] Status changed from uninitialized to yellow - Waiting for Elasticsearch

log [095900689] [info][status][pluginkbn_vislib_vis_types] Status changed from uninitialized to green - Ready

log [095900717] [info][status][pluginmarkdown_vis] Status changed from uninitialized to green - Ready

log [095900721] [info][status][pluginmetric_vis] Status changed from uninitialized to green - Ready

log [095900814] [info][status][pluginspyModes] Status changed from uninitialized to green - Ready

log [095900934] [info][status][pluginstatusPage] Status changed from uninitialized to green - Ready

Page of 18 71 Page of 18 71

log [095900941] [info][status][plugintable_vis] Status changed from uninitialized to green - Ready

log [095901067] [info][listening] Server running at http00005601

log [095906083] [info][status][pluginelasticsearch] Status changed from yellow to yellow - No existing Kibana index found

log [095908942] [info][status][pluginelasticsearch] Status changed from yellow to green - Kibana index ready

Kibana13 is13 now13 started13

Broadcast13 Bluemix13 app13 log13 records13 Bluemix13 apps13 (which13 are13 shown13 in13 the13 US13 South13 Bluemix13 region13 of13 the13 System13 Architecture13 diagram)13 will13 produce13 logs13 in13 a13 syslog13 rfc542413 format13 using13 a13 Bluemix13 tool13 called13 Loggregator13

For13 Logstash13 to13 receive13 these13 logs13 each13 Bluemix13 app13 needs13 to13 broadcast13 these13 logs13 To13 do13 that13 run13 the13 following13 commands13 on13 each13 Bluemix13 app13 using13 the13 public13 IP13 address13 of13 the13 ELK13 virtual13 server13 the13 ELK13 tag13 to13 trace13 the13 process13 and13 insert13 the13 name13 of13 each13 app13 in13 turn13

cf cups elk -l syslogltpublicIPaddress5000 cf bind-service ltmyappgt elk

cf restage ltmyappgt

Install13 Logstash13 To13 install13 Logstash13 as13 root13 user13 create13 a13 directory13 for13 the13 installaon13 and13 untar13 the13 installaon13 file13 there13

13 13 [rootelkforcase]$ cd home

[rootelkforcase]$ mkdir logstash

[rootelkforcase]$ cp homeibmcloudlogstash-233targz homelogstash

[rootelkforcase]$ cd logstash

[rootelkforcase]$ gunzip logstash-233targz

[rootelkforcase]$ untar logstash-233tar

Write13 Logstash13 configura0on13 Logstash13 requires13 a13 configuraon13 file13 to13 be13 wricen13 to13 receive13 transform13 and13 output13 the13 log13 records13 to13 Elascsearch13 This13 secon13 explains13 the13 basic13 requirements13 of13 processing13 Bluemix13 app13 log13 records13 into13 Elascsearch13 (more13 complex13 configuraons13 are13 detailed13 later13 in13 this13 document13 for13 the13 integraons13 with13 third-shy‐party13 tools)13

Create13 a13 conf13 file13 as13 root13 user13

[rootelkforcase]$ cd homelogstashlogstash-233

[rootelkforcase]$ mkdir config

Page of 19 71 Page of 19 71

[rootelkforcase]$ cd config

[rootelkforcase]$ touch loggregatorconf

Edit13 the13 loggregator13 file13 with13 the13 following13 contents13

13 Input13 secon13 defines13 which13 log13 records13 to13 receive13

13 port13 and13 type13 reflect13 method13 selected13 for13 Cloud13 Foundry13 to13 broadcast13 the13 records13

input

tcp

port =gt 5000

type =gt syslog

13 Filter13 secon13 ndash13 transforms13 the13 data13

filter

Add a field to identify the records as Cloud Foundry based

mutate

add_field =gt format =gt cf

13 Output13 secon13 ndash13 sends13 the13 data13 to13 its13 desnaon(s)13

output

13 Use13 the13 elascsearch13 plugin13 to13 send13 the13 data13 to13 the13 elascsearch13 instance13

13 elasticsearch

hosts =gt localhost

stdout codec =gt rubydebug

13 End13 of13 Config13

Save13 the13 file13 Verify13 the13 format13 of13 the13 file13 is13 acceptable13 by13 running13 the13 following13 commands13

[rootelkforcase]$ cd homelogstashlogstash-233

[rootelkforcase]$ binlogstash agent --configtest -f configloggregatorconf

Page of 20 71 Page of 20 71

Configuraon13 is13 OK13

Start13 Logstash13 Aaer13 you13 have13 wricen13 and13 verified13 the13 Logstash13 configuraon13 you13 can13 start13 Logstash13 to13 process13 records13 from13 the13 Bluemix13 apps13 (via13 Cloud13 Foundry)13 Run13 the13 following13 as13 root13 user13

[rootelkforcase]$ cd homelogstashlogstash-233

[rootelkforcase]$ binlogstash agent -f configloggregatorconf

Logstash13 is13 now13 started13

Monitor13 Logstash13 processing13 Once13 Logstash13 is13 running13 and13 log13 records13 are13 being13 produced13 by13 the13 Bluemix13 apps13 you13 can13 monitor13 progress13 in13 the13 same13 window13 that13 was13 used13 to13 start13 Logstash13 Output13 is13 similar13 to13 the13 following13

13

The13 log13 records13 being13 broadcast13 from13 the13 Bluemix13 apps13 are13 now13 sent13 to13 Elascsearch13 from13 the13 Logstash13 output13 configuraon13 Records13 are13 sent13 in13 real13 me13 If13 Logstash13 is13 not13 running13 while13 log13 records13 are13 being13 produced13 then13 these13 records13 will13 not13 be13 processed13 by13 Logstash13 and13 will13 not13 be13 stored13 in13 Elascsearch13

Page of 21 71 Page of 21 71

Use13 ELK13 The13 log13 records13 from13 the13 Bluemix13 apps13 are13 now13 being13 stored13 and13 indexed13 by13 the13 Elascsearch13 component13 This13 data13 can13 be13 accessed13 via13 the13 Kibana13 applicaon13

Kibana13 Discover13 page13 Using13 a13 web13 browser13 either13 directly13 from13 a13 client13 or13 using13 the13 VNC13 client13 access13 Kibana13 using13 the13 publicIPaddress13 of13 the13 ELK13 virtual13 server13

hcpltpublicIPaddressgt560113

No13 userpassword13 credenals13 are13 required13

Kibana13 defaults13 to13 a13 search13 of13 all13 records13 ()13 in13 the13 last13 1513 minutes13 and13 returns13 the13 results13

13

More13 complex13 searches13 are13 possible13 by13 changing13 the13 me13 filter13 and13 supplying13 search13 queries13

Change13 the13 0me13 filter13 To13 change13 the13 me13 ranges13 click13 the13 clock13 symbol13 in13 the13 top-shy‐right13 of13 the13 Kibana13 Discover13 page13

13

You13 will13 see13 the13 following13 opons13

13

Page of 22 71 Page of 22 71

A13 variety13 of13 quick13 me13 ranges13 can13 be13 selected13 or13 you13 can13 switch13 to13 relave13 or13 absolute13 me13 ranges13

Quick13 me13 ranges13 allow13 switching13 between13 last13 X13 minutes13 to13 last13 Y13 hours13 or13 Z13 days13 etc13

Relave13 me13 ranges13 allow13 more13 specific13 versions13 of13 the13 Quick13 opons13 from13 specific13 minuteshoursdays13 to13 now13

13

Absolute13 me13 ranges13 specify13 the13 start13 and13 end13 mes13 for13 the13 searches13

13

Custom13 search13 queries13 You13 can13 use13 the13 text13 window13 of13 the13 Discover13 page13 to13 search13 for13 matches13 within13 the13 log13 records13 You13 can13 use13 Apache13 Lucene13 or13 Elascsearch13 query13 DSL13 syntax13

13

Page of 23 71 Page of 23 71

Saved13 searches13 You13 can13 save13 any13 custom13 search13 syntax13 and13 choice13 of13 me13 range13 for13 future13 ease13 of13 use13 Any13 relave13 or13 absolute13 me13 frame13 is13 saved13 as13 part13 of13 the13 search13 13

To13 save13 a13 search13

1 Set13 up13 and13 run13 the13 required13 syntax13 and13 me13 frame13

13

2 Click13 the13 Save13 Search13 icon13

13

3 Type13 in13 a13 name13 for13 the13 saved13 search13

Page of 24 71 Page of 24 71

13

7 To13 delete13 or13 update13 saved13 searches13 click13 13 manage13 saved13 searches13

13

Use13 the13 edit13 icons13 to13 make13 any13 changes13

Page of 26 71 Page of 26 71

13

Build13 simple13 Kibana13 histograms13 Kibana13 allows13 you13 to13 build13 charts13 and13 dashboards13 based13 on13 the13 search13 capability13 offered13 by13 the13 Discover13 page13 Saved13 searches13 can13 be13 used13 as13 the13 basis13 for13 charts13 and13 dashboards13 built13 using13 those13 charts13 To13 build13 a13 simple13 histogram13

1 Select13 the13 Visualize13 page13

13

2 Select13 Ver0cal13 Bar13 Chart13

13

3 To13 use13 a13 saved13 search13 click13 From13 a13 saved13 search13

Page of 27 71 Page of 27 71

13

4 Select13 a13 previously13 saved13 search13

13

5 Select13 to13 edit13 the13 x-shy‐axis

6 Edit13 the13 fields13 with13 informaon13 about13 the13 aggregaon13 field13 interval13 and13 custom13 label13

Page of 28 71 Page of 28 71

13

Page of 29 71 Page of 29 71

7 Click13 the13 green13 arrow13 The13 histogram13 is13 created13 and13 run13 based13 on13 the13 saved13 search

13

8 To13 save13 the13 histogram13 click13 the13 Save13 Visualiza0on13 icon13 and13 add13 a13 tle13

13

13

9 Click13 Save13

10 To13 view13 the13 visualisaon13 click13 the13 Dashboard13 page13

Page of 30 71 Page of 30 71

13

11 Click13 the13 Add13 Visualiza0on13 icon13

13

12 Select13 the13 visiualizaon13 that13 you13 just13 created13

13

Page of 31 71 Page of 31 71

13

13 To13 save13 the13 dashboard13 click13 the13 Save13 Dashboard13 icon13

Page of 32 71 Page of 32 71

13

14 Add13 a13 tle13 and13 select13 whether13 to13 store13 the13 selected13 me13 frame13 with13 the13 dashboard13

13

15 Click13 Save13

16 You13 can13 edit13 the13 shape13 and13 number13 of13 charts13 and13 save13 the13 dashboard13

17 To13 re-shy‐run13 the13 dashboard13 select13 the13 Dashboard13 page13 and13 click13 the13 Load13 Saved13 Dashboard13 icon13

Page of 33 71 Page of 33 71

13

18 Select13 the13 saved13 dashboard13 from13 the13 list13

13

19 The13 dashboard13 runs13

13

Page of 34 71 Page of 34 71

Page of 35 71 Page of 35 71

Aler0ng13 with13 ELK13 ELK13 examines13 the13 log13 records13 in13 process13 to13 send13 alerts13 to13 third-shy‐party13 alerng13 event13 management13 and13 social13 media13 applicaons13 As13 shown13 previously13 in13 the13 ELK13 architecture13 the13 ELK13 strategy13 for13 alerng13 is13 to13 send13 alerts13 from13 the13 Logstash13 component13 directly13 Therefore13 log13 records13 and13 alerts13 are13 sent13 to13 both13 Elascsearch13 and13 third-shy‐party13 tools13 at13 the13 same13 instant13 using13 the13 output13 secon13 of13 the13 Logstash13 configuraon13 Any13 alerts13 are13 generated13 during13 the13 filter13 secon13 processing13 of13 the13 Logstash13 configuraon13

Create13 alerts13 in13 Logstash13 The13 Logstash13 filter13 secon13 is13 used13 to13 trigger13 alerts13 based13 on13 the13 log13 records13 being13 processed13 For13 example13 log13 records13 can13 be13 checked13 for13 errors13 warnings13 crashed13 or13 stopped13 messages13 and13 generate13 an13 alert13

Here13 is13 an13 example13 Logstash13 filter13 code13

13 if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

The13 specific13 tags13 are13 set13 against13 the13 log13 records13 to13 trigger13 the13 alert13 later13 in13 the13 configuraon13 in13 the13 output13 secon13

Control13 the13 number13 of13 generated13 alerts13 using13 the13 Logstash13 throcle13 plugin13 so13 that13 either13 a13 lot13 of13 instances13 in13 a13 fixed13 me13 period13 are13 required13 before13 an13 alert13 is13 sent13 or13 a13 specific13 number13 of13 instances13 triggers13 the13 alert13

Page of 36 71 Page of 36 71

Edit13 the13 plugin13 parameters13 to13 meet13 system13 requirements13

13 13 13 13 13 if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 10

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle13

after_count =gt 20

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 30

period =gt 1200

key =gt servicename_alarm_warning

Page of 37 71 Page of 37 71

add_tag =gt servicename_throttled

13 13 13 13 13 13 13 13 13

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 40

period =gt 1500

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

Enrich13 alerts13 with13 message13 detail13 Some13 of13 the13 third-shy‐party13 integraons13 benefit13 from13 having13 components13 of13 the13 raw13 log13 record13 sent13 in13 addion13 to13 the13 raw13 record13 or13 fields13 in13 the13 record13 edited13 transformed13 or13 translated13

Grok13 Logstash13 allows13 the13 log13 record13 to13 be13 broken13 up13 into13 logical13 secons13 and13 each13 secon13 stored13 in13 a13 new13 parameter13 using13 the13 grok13 parameter13 in13 the13 filter13 secon13 Here13 is13 an13 example13 grok13

13 grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

To13 build13 the13 grok13 statement13 Logstash13 offers13 an13 online13 tool13 at13 hcpsgrokdebugherokuappcom13

Translate13 If13 specific13 fields13 in13 the13 log13 record13 are13 coded13 (for13 example13 the13 Bluemix13 app13 name13 is13 oaen13 not13 sent13 in13 the13 syslog13 messages13 instead13 the13 GUID13 is13 sent13 which13 is13 not13 useful)13 then13 the13 filter13 secon13 translate13 plugin13 can13 be13 used13 to13 add13 new13 fields13 by13 holding13 the13 decoded13 versions13 of13 those13 values13

13 translate

field =gt data1

destination =gt data1

override =gt true

Page of 38 71 Page of 38 71

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

dc50f6e4-c052-436a-bf27-95ba392c4228 monverifyservicemanagement

0c677e82-5bfc-4e38-9155-e416249e3ee6 ms-catalog-postpyretic-pomiculture

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

However13 the13 translate13 plugin13 is13 not13 included13 with13 the13 Logstash13 installaon13 In13 the13 next13 secon13 I13 will13 13 describe13 how13 to13 add13 new13 plugins13 to13 the13 Logstash13 instance13

Add13 custom13 plugins13 to13 Logstash13 As13 an13 open13 source13 tool13 Logstash13 encourages13 users13 to13 develop13 content13 for13 all13 users13 to13 share13 Some13 of13 the13 plugins13 that13 were13 created13 by13 our13 community13 include13 a13 translaon13 plugin13 an13 output13 Slack13 plugin13 and13 an13 output13 syslog13 plugin13 Letrsquos13 look13 at13 how13 to13 add13 these13 plugins13 to13 your13 environment13

Filter13 translate13 plugin13 To13 add13 the13 translate13 plugin13

1 Download13 the13 plugin13

2 Copy13 the13 plugin13 file13 logstash-shy‐filter-shy‐translate-shy‐214gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-filter-translate 214

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Output13 Slack13 plugin13 To13 add13 the13 Slack13 plugin13

Page of 39 71 Page of 39 71

1 Download13 the13 plugin13 13

2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐slack-shy‐014gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-output-slack 014

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Output13 syslog13 plugin13 To13 add13 the13 syslog13 plugin13

1 Download13 the13 plugin13 13

2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐syslog-shy‐214gem13 to13 the13 ELK13 Virtual13 Server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-output-syslog 214

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Integrate13 Logstash13 with13 Slack13 To13 send13 generated13 alerts13 to13 Slack13 complete13 these13 steps13

1 Create13 a13 Slack13 integraon13 13

2 Use13 the13 Slack13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 Slack13 integraon13

113 Login13 to13 Slack13 and13 go13 to13 the13 Apps13 page13 13

Page of 40 71 Page of 40 71

13

313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13

13

A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13

13

Page of 42 71 Page of 42 71

The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13

13

The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13

13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

slack

url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5

channel =gt cloud-operations-mon

username =gt nick_cawood

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13

Page of 43 71 Page of 43 71

Page of 44 71 Page of 44 71

In13 Slack13 the13 alerts13 look13 like13 this13

13

Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13

1 Create13 a13 PagerDuty13 integraon13 13

2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13

1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13

13

2 Click13 Add13 New13 Service13

13

313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13

Page of 45 71 Page of 45 71

13

413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13

513 Click13 Add13 Service13

13

The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13

Page of 46 71 Page of 46 71

13

The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt be7eb1ea9c224018923e818d3e891d0b

incident_key =gt logstashhosttype

Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13

Page of 47 71 Page of 47 71

13

Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13

1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13

2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13

3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13

4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13

13 13 http

url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo

http_method =gt post

format =gt json

For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13

13 13 syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

Page of 48 71 Page of 48 71

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13

13

The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13

if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2

This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13

Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13

Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13

Logstash13 severity Shown13 in13 Omnibus13 as

alert Indeterminate

crical Warning

debug 7

emergency Clear

error Minor

informaonal 6

noce Crical

warning Major

Page of 49 71 Page of 49 71

13

Page of 50 71 Page of 50 71

Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13

input

tcp

port =gt 5000

type =gt syslog

filter

mutate

add_field =gt format =gt cf

grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

translate

field =gt data1

destination =gt data1

override =gt true

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

=== Check Alarm Criteria ===

Standard Key Words

if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

=== Alarm Exclusions ===

if [message] =~ (i)sometexthere

mutate remove_tag =gt servicename_alarm

Throttle alarms so we do not hit PD rate limits

Page of 52 71 Page of 52 71

if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 5

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle

after_count =gt 5

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 5

period =gt 1200

key =gt servicename_alarm_warning

add_tag =gt servicename_throttled

Page of 53 71 Page of 53 71

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

output

elasticsearch

hosts =gt localhost

stdout codec =gt rubydebug

if servicename_alarm_err in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

Page of 54 71 Page of 54 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 55 71 Page of 55 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

Page of 56 71 Page of 56 71

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert err message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_error in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

Page of 57 71 Page of 57 71

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 58 71 Page of 58 71

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 59 71 Page of 59 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert error message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

Page of 60 71 Page of 60 71

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

Page of 61 71 Page of 61 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 62 71 Page of 62 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt x

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert fatal message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

Page of 63 71 Page of 63 71

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

Page of 64 71 Page of 64 71

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 65 71 Page of 65 71

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt heavy_exclamation_mark

Page of 66 71 Page of 66 71

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert WARNING message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 67 71 Page of 67 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

Page of 68 71 Page of 68 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 69 71 Page of 69 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

Page of 70 71 Page of 70 71

Page of 71 71 Page of 71 71

Page 13: How to integrate the Elasticsearch Logstash Kibana (ELK ... · How to integrate the Elasticsearch Logstash Kibana (ELK) log analytics stack into IBM Bluemix Nick Cawood IBM Cloud

Copy13 files13 to13 the13 virtual13 server13 To13 move13 installaon13 files13 to13 the13 server13 go13 to13 the13 UNIX13 or13 Linux13 window13 you13 used13 to13 create13 the13 SSH13 key13 and13 the13 directory13 where13 the13 ltuniquekeynamegt13 files13 exist13 Move13 the13 installaon13 files13 to13 the13 same13 locaon13

Run13 the13 following13 command13 referencing13 the13 SSH13 keyname13 the13 public13 IP13 address13 Bluemix13 allocated13 for13 the13 virtual13 server13 and13 the13 password13 for13 the13 SSH13 key13

[userserver ~]$ scp ndashi ltuniquekeynamegt ltINSTALLATIONFILENAMEgt ibmcloudltpublicIPaddressgthomeibmcloud

Enter13 the13 passphrase13 for13 key13 cloudkey313

This13 copies13 the13 file13 using13 the13 ibmcloud13 user13 to13 the13 homeibmcloud13 directory13 on13 the13 virtual13 server13

Install13 and13 access13 the13 virtual13 server13 via13 VNC13 server13 Using13 the13 ssh13 command13 to13 access13 the13 virtual13 server13 is13 not13 the13 most13 stable13 method13 and13 does13 not13 offer13 X-shy‐Windows13 support13 Therefore13 we13 suggest13 that13 you13 use13 a13 VNC13 client13 to13 access13 the13 virtual13 server13

Installing13 a13 VNC13 client13 should13 be13 done13 as13 the13 root13 user13 Use13 the13 yum13 command13 as13 follows13

[ibmcloudelkforcase ~]$ yum groupinstall GNOME Desktop

[ibmcloudelkforcase ~]$ yum install tigervnc-server

These13 commands13 first13 install13 a13 desktop13 style13 OS13 user13 interface13 and13 then13 a13 VNC13 client13 to13 remotely13 access13 the13 desktop13 style13 OS13

Then13 as13 either13 the13 root13 user13 or13 another13 user13 (example13 below13 uses13 elk)13 run13 the13 following13 command13 to13 create13 the13 VNC13 client13 connecon13

[elkelkforcase ~]$ vncserver

New elkforcase2 (elk) desktop is elkforcase2

Starting applications specified in homeelkvncxstartup

Log file is homeelkvncelkforcase2log

[elkelkforcase ~]$

Install13 a13 VNC13 client13 on13 the13 client13 machine13 used13 to13 access13 the13 Internet13 and13 (in13 the13 example13 above)13 use13 the13 command13 ltpublicIPadddressgt213 and13 the13 password13 for13 ELK13 to13 access13 the13 virtual13 server13

Using13 a13 VNC13 client13 means13 that13 you13 can13 leave13 long13 duraon13 installaons13 running13 without13 fear13 of13 the13 terminal13 losing13 connecon13 and13 the13 installaons13 failing13

Install13 the13 ELK13 stack13 Download13 the13 source13 soaware13 code13 for13 the13 ELK13 stack13 from13 the13 ELK13 website13 13

At13 the13 me13 of13 wring13 the13 downloaded13 soaware13 files13 were13

Page of 13 71 Page of 13 71

13

Transfer13 the13 files13 to13 the13 virtual13 server13 (using13 the13 scp13 method13 detailed13 above)13

Install13 Elas0csearch13 As13 a13 root13 user13 create13 a13 non-shy‐root13 user13 with13 the13 adduser13 command13

Create13 a13 directory13 to13 install13 Elascsearch13 into13 and13 untar13 the13 contents13 of13 the13 Elascsearch13 install13 file13 into13 that13 directory13

[elkelkforcase]$ cd home

[elkelkforcase]$ mkdir elasticsearch

[elkelkforcase]$ cp homeibmcloudelasticsearch-233targz homeelasticsearch

[elkelkforcase]$ cd elasticsearch

[elkelkforcase]$ gunzip elasticsearch-233targz

[elkelkforcase]$ untar elasticsearch-233tar

Elascsearch13 is13 now13 installed13

Start13 Elas0csearch13 To13 start13 Elascsearch13 go13 to13 the13 bin13 directory13 as13 the13 non-shy‐root13 user13 when13 you13 installed13 it13 and13 run13 the13 elascsearch13 executable13

[elkelkforcase]$ cd homeelasticsearchelasticsearch-233bin

[elkelkforcase]$ elasticsearch amp

[2] 22211

[elkelkforcase]$ [2016-07-05 133857730][INFO ][node] [Arsenal] version[233] pid[22211] build[218bdf12016-05-17T154004Z]

[2016-07-05 133857740][INFO ][node] [Arsenal] initializing

[2016-07-05 133858984][INFO ][plugins] [Arsenal] modules [reindex lang-expression lang-groovy] plugins [] sites []

[2016-07-05 133859146][INFO ][env] [Arsenal] using [1] data paths mounts [[ (rootfs)]] net usable_space [155gb] net total_space [199gb] spins [unknown] types [rootfs]

[2016-07-05 133859146][INFO ][env] [Arsenal] heap size [10156mb] compressed ordinary object pointers [true]

Page of 14 71 Page of 14 71

[2016-07-05 133859147][WARN ][env] [Arsenal] max file descriptors [4096] for elasticsearch process likely too low consider increasing to at least [65536]

[2016-07-05 133903526][INFO ][node] [Arsenal] initialized

[2016-07-05 133903526][INFO ][node] [Arsenal] starting

To13 test13 that13 Elascsearch13 is13 running13 correctly13 open13 the13 browser13 in13 the13 VNC13 client13 and13 run13 localhost920013

13

Install13 Kibana13 ELK13 recommends13 installing13 and13 running13 Kibana13 as13 root13 13

1 As13 root13 user13 create13 an13 installaon13 directory13 and13 untar13 the13 installaon13 file13

[rootelkforcase]$ cd home

[rootelkforcase]$ mkdir kibana

[rootelkforcase]$ cp homeibmcloudkibana-451-linux-x64targz homekibana

[rootelkforcase]$ cd kibana

[rootelkforcase]$ gunzip kibana-451-linux-x64targz

[rootelkforcase]$ untar kibana-451-linux-x64tar

2 Modify13 the13 manifestyml13 configuraon13 file13 Go13 to13 the13 kibana-shy‐451-shy‐linux-shy‐x6413 directory13

[rootelkforcase]$ cd homekibanakibana-451-linux-x64

3 Edit13 the13 manifestyml13 file13 so13 the13 contents13 are13

applicaons13

-shy‐ name13 kibana-shy‐in-shy‐bluemix13

memory13 128M13

host13 kibana-shy‐in-shy‐bluemix13

buildpack13 hcpsgithubcomcloudfoundry-shy‐communitynginx-shy‐buildpackgit13

Page of 15 71 Page of 15 71

4 Edit13 the13 configkibanayml13 file13 so13 the13 contents13 are13

Kibana is served by a back end server This controls which port to use

port 5601

The host to bind the server to

host 0000

If you are running kibana behind a proxy and want to mount it at a path

specify that path here The basePath cant end in a slash

serverbasePath

The maximum payload size in bytes on incoming server requests

servermaxPayloadBytes 1048576

The Elasticsearch instance to use for all your queries

elasticsearchurl httplocalhost9200

preserve_elasticsearch_host true will send the hostname specified in `elasticsearch` If you set it to false

then the host you use to connect to this Kibana instance will be sent

elasticsearchpreserveHost true

Kibana uses an index in Elasticsearch to store saved searches visualizations

and dashboards It will create a new index if it doesnt already exist

kibanaindex kibana

The default application to load

kibanadefaultAppId discover

If your Elasticsearch is protected with basic auth these are the user credentials

used by the Kibana server to perform maintenance on the kibana_index at startup Your

Kibana users will still need to authenticate with Elasticsearch (which is proxied through

the Kibana server)

Page of 16 71 Page of 16 71

elasticsearchusername user

elasticsearchpassword pass

SSL for outgoing requests from the Kibana Server to the browser (PEM formatted)

serversslcert pathtoyourservercrt

serversslkey pathtoyourserverkey

Optional setting to validate that your Elasticsearch backend uses the same key files (PEM formatted)

elasticsearchsslcert pathtoyourclientcrt

elasticsearchsslkey pathtoyourclientkey

If you need to provide a CA certificate for your Elasticsearch instance put

the path of the pem file here

elasticsearchsslca pathtoyourCApem

Set to false to have a complete disregard for the validity of the SSL

certificate

elasticsearchsslverify true

Time in milliseconds to wait for elasticsearch to respond to pings defaults to

request_timeout setting

elasticsearchpingTimeout 1500

Time in milliseconds to wait for responses from the back end or elasticsearch

This must be gt 0

elasticsearchrequestTimeout 30000

Time in milliseconds for Elasticsearch to wait for responses from shards

Set to 0 to disable

elasticsearchshardTimeout 0

Time in milliseconds to wait for Elasticsearch at Kibana startup before retrying

elasticsearchstartupTimeout 5000

elasticsearchstartupTimeout 30000

Page of 17 71 Page of 17 71

Set the path to where you would like the process id file to be created

pidfile varrunkibanapid

If you would like to send the log output to a file you can set the path below

loggingdest stdout

Set this to true to suppress all logging output

loggingsilent false

Set this to true to suppress all logging output except for error messages

loggingquiet false

Set this to true to log all events including system usage information and all requests

loggingverbose false

Kibana13 is13 now13 installed13

Start13 Kibana13 To13 start13 Kibana13 go13 to13 the13 bin13 directory13 and13 run13 the13 kibana13 executable13

[rootelkforcase]$

[rootelkforcase]$ cd homekibanakibana-451-linux-x64bin

[rootelkforcase]$ kibana amp

[1] 22283

[rootelkforcase] log [095900542] [info][status][pluginkibana] Status changed from uninitialized to green - Ready

log [095900600] [info][status][pluginelasticsearch] Status changed from uninitialized to yellow - Waiting for Elasticsearch

log [095900689] [info][status][pluginkbn_vislib_vis_types] Status changed from uninitialized to green - Ready

log [095900717] [info][status][pluginmarkdown_vis] Status changed from uninitialized to green - Ready

log [095900721] [info][status][pluginmetric_vis] Status changed from uninitialized to green - Ready

log [095900814] [info][status][pluginspyModes] Status changed from uninitialized to green - Ready

log [095900934] [info][status][pluginstatusPage] Status changed from uninitialized to green - Ready

Page of 18 71 Page of 18 71

log [095900941] [info][status][plugintable_vis] Status changed from uninitialized to green - Ready

log [095901067] [info][listening] Server running at http00005601

log [095906083] [info][status][pluginelasticsearch] Status changed from yellow to yellow - No existing Kibana index found

log [095908942] [info][status][pluginelasticsearch] Status changed from yellow to green - Kibana index ready

Kibana13 is13 now13 started13

Broadcast13 Bluemix13 app13 log13 records13 Bluemix13 apps13 (which13 are13 shown13 in13 the13 US13 South13 Bluemix13 region13 of13 the13 System13 Architecture13 diagram)13 will13 produce13 logs13 in13 a13 syslog13 rfc542413 format13 using13 a13 Bluemix13 tool13 called13 Loggregator13

For13 Logstash13 to13 receive13 these13 logs13 each13 Bluemix13 app13 needs13 to13 broadcast13 these13 logs13 To13 do13 that13 run13 the13 following13 commands13 on13 each13 Bluemix13 app13 using13 the13 public13 IP13 address13 of13 the13 ELK13 virtual13 server13 the13 ELK13 tag13 to13 trace13 the13 process13 and13 insert13 the13 name13 of13 each13 app13 in13 turn13

cf cups elk -l syslogltpublicIPaddress5000 cf bind-service ltmyappgt elk

cf restage ltmyappgt

Install13 Logstash13 To13 install13 Logstash13 as13 root13 user13 create13 a13 directory13 for13 the13 installaon13 and13 untar13 the13 installaon13 file13 there13

13 13 [rootelkforcase]$ cd home

[rootelkforcase]$ mkdir logstash

[rootelkforcase]$ cp homeibmcloudlogstash-233targz homelogstash

[rootelkforcase]$ cd logstash

[rootelkforcase]$ gunzip logstash-233targz

[rootelkforcase]$ untar logstash-233tar

Write13 Logstash13 configura0on13 Logstash13 requires13 a13 configuraon13 file13 to13 be13 wricen13 to13 receive13 transform13 and13 output13 the13 log13 records13 to13 Elascsearch13 This13 secon13 explains13 the13 basic13 requirements13 of13 processing13 Bluemix13 app13 log13 records13 into13 Elascsearch13 (more13 complex13 configuraons13 are13 detailed13 later13 in13 this13 document13 for13 the13 integraons13 with13 third-shy‐party13 tools)13

Create13 a13 conf13 file13 as13 root13 user13

[rootelkforcase]$ cd homelogstashlogstash-233

[rootelkforcase]$ mkdir config

Page of 19 71 Page of 19 71

[rootelkforcase]$ cd config

[rootelkforcase]$ touch loggregatorconf

Edit13 the13 loggregator13 file13 with13 the13 following13 contents13

13 Input13 secon13 defines13 which13 log13 records13 to13 receive13

13 port13 and13 type13 reflect13 method13 selected13 for13 Cloud13 Foundry13 to13 broadcast13 the13 records13

input

tcp

port =gt 5000

type =gt syslog

13 Filter13 secon13 ndash13 transforms13 the13 data13

filter

Add a field to identify the records as Cloud Foundry based

mutate

add_field =gt format =gt cf

13 Output13 secon13 ndash13 sends13 the13 data13 to13 its13 desnaon(s)13

output

13 Use13 the13 elascsearch13 plugin13 to13 send13 the13 data13 to13 the13 elascsearch13 instance13

13 elasticsearch

hosts =gt localhost

stdout codec =gt rubydebug

13 End13 of13 Config13

Save13 the13 file13 Verify13 the13 format13 of13 the13 file13 is13 acceptable13 by13 running13 the13 following13 commands13

[rootelkforcase]$ cd homelogstashlogstash-233

[rootelkforcase]$ binlogstash agent --configtest -f configloggregatorconf

Page of 20 71 Page of 20 71

Configuraon13 is13 OK13

Start13 Logstash13 Aaer13 you13 have13 wricen13 and13 verified13 the13 Logstash13 configuraon13 you13 can13 start13 Logstash13 to13 process13 records13 from13 the13 Bluemix13 apps13 (via13 Cloud13 Foundry)13 Run13 the13 following13 as13 root13 user13

[rootelkforcase]$ cd homelogstashlogstash-233

[rootelkforcase]$ binlogstash agent -f configloggregatorconf

Logstash13 is13 now13 started13

Monitor13 Logstash13 processing13 Once13 Logstash13 is13 running13 and13 log13 records13 are13 being13 produced13 by13 the13 Bluemix13 apps13 you13 can13 monitor13 progress13 in13 the13 same13 window13 that13 was13 used13 to13 start13 Logstash13 Output13 is13 similar13 to13 the13 following13

13

The13 log13 records13 being13 broadcast13 from13 the13 Bluemix13 apps13 are13 now13 sent13 to13 Elascsearch13 from13 the13 Logstash13 output13 configuraon13 Records13 are13 sent13 in13 real13 me13 If13 Logstash13 is13 not13 running13 while13 log13 records13 are13 being13 produced13 then13 these13 records13 will13 not13 be13 processed13 by13 Logstash13 and13 will13 not13 be13 stored13 in13 Elascsearch13

Page of 21 71 Page of 21 71

Use13 ELK13 The13 log13 records13 from13 the13 Bluemix13 apps13 are13 now13 being13 stored13 and13 indexed13 by13 the13 Elascsearch13 component13 This13 data13 can13 be13 accessed13 via13 the13 Kibana13 applicaon13

Kibana13 Discover13 page13 Using13 a13 web13 browser13 either13 directly13 from13 a13 client13 or13 using13 the13 VNC13 client13 access13 Kibana13 using13 the13 publicIPaddress13 of13 the13 ELK13 virtual13 server13

hcpltpublicIPaddressgt560113

No13 userpassword13 credenals13 are13 required13

Kibana13 defaults13 to13 a13 search13 of13 all13 records13 ()13 in13 the13 last13 1513 minutes13 and13 returns13 the13 results13

13

More13 complex13 searches13 are13 possible13 by13 changing13 the13 me13 filter13 and13 supplying13 search13 queries13

Change13 the13 0me13 filter13 To13 change13 the13 me13 ranges13 click13 the13 clock13 symbol13 in13 the13 top-shy‐right13 of13 the13 Kibana13 Discover13 page13

13

You13 will13 see13 the13 following13 opons13

13

Page of 22 71 Page of 22 71

A13 variety13 of13 quick13 me13 ranges13 can13 be13 selected13 or13 you13 can13 switch13 to13 relave13 or13 absolute13 me13 ranges13

Quick13 me13 ranges13 allow13 switching13 between13 last13 X13 minutes13 to13 last13 Y13 hours13 or13 Z13 days13 etc13

Relave13 me13 ranges13 allow13 more13 specific13 versions13 of13 the13 Quick13 opons13 from13 specific13 minuteshoursdays13 to13 now13

13

Absolute13 me13 ranges13 specify13 the13 start13 and13 end13 mes13 for13 the13 searches13

13

Custom13 search13 queries13 You13 can13 use13 the13 text13 window13 of13 the13 Discover13 page13 to13 search13 for13 matches13 within13 the13 log13 records13 You13 can13 use13 Apache13 Lucene13 or13 Elascsearch13 query13 DSL13 syntax13

13

Page of 23 71 Page of 23 71

Saved13 searches13 You13 can13 save13 any13 custom13 search13 syntax13 and13 choice13 of13 me13 range13 for13 future13 ease13 of13 use13 Any13 relave13 or13 absolute13 me13 frame13 is13 saved13 as13 part13 of13 the13 search13 13

To13 save13 a13 search13

1 Set13 up13 and13 run13 the13 required13 syntax13 and13 me13 frame13

13

2 Click13 the13 Save13 Search13 icon13

13

3 Type13 in13 a13 name13 for13 the13 saved13 search13

Page of 24 71 Page of 24 71

13

7 To13 delete13 or13 update13 saved13 searches13 click13 13 manage13 saved13 searches13

13

Use13 the13 edit13 icons13 to13 make13 any13 changes13

Page of 26 71 Page of 26 71

13

Build13 simple13 Kibana13 histograms13 Kibana13 allows13 you13 to13 build13 charts13 and13 dashboards13 based13 on13 the13 search13 capability13 offered13 by13 the13 Discover13 page13 Saved13 searches13 can13 be13 used13 as13 the13 basis13 for13 charts13 and13 dashboards13 built13 using13 those13 charts13 To13 build13 a13 simple13 histogram13

1 Select13 the13 Visualize13 page13

13

2 Select13 Ver0cal13 Bar13 Chart13

13

3 To13 use13 a13 saved13 search13 click13 From13 a13 saved13 search13

Page of 27 71 Page of 27 71

13

4 Select13 a13 previously13 saved13 search13

13

5 Select13 to13 edit13 the13 x-shy‐axis

6 Edit13 the13 fields13 with13 informaon13 about13 the13 aggregaon13 field13 interval13 and13 custom13 label13

Page of 28 71 Page of 28 71

13

Page of 29 71 Page of 29 71

7 Click13 the13 green13 arrow13 The13 histogram13 is13 created13 and13 run13 based13 on13 the13 saved13 search

13

8 To13 save13 the13 histogram13 click13 the13 Save13 Visualiza0on13 icon13 and13 add13 a13 tle13

13

13

9 Click13 Save13

10 To13 view13 the13 visualisaon13 click13 the13 Dashboard13 page13

Page of 30 71 Page of 30 71

13

11 Click13 the13 Add13 Visualiza0on13 icon13

13

12 Select13 the13 visiualizaon13 that13 you13 just13 created13

13

Page of 31 71 Page of 31 71

13

13 To13 save13 the13 dashboard13 click13 the13 Save13 Dashboard13 icon13

Page of 32 71 Page of 32 71

13

14 Add13 a13 tle13 and13 select13 whether13 to13 store13 the13 selected13 me13 frame13 with13 the13 dashboard13

13

15 Click13 Save13

16 You13 can13 edit13 the13 shape13 and13 number13 of13 charts13 and13 save13 the13 dashboard13

17 To13 re-shy‐run13 the13 dashboard13 select13 the13 Dashboard13 page13 and13 click13 the13 Load13 Saved13 Dashboard13 icon13

Page of 33 71 Page of 33 71

13

18 Select13 the13 saved13 dashboard13 from13 the13 list13

13

19 The13 dashboard13 runs13

13

Page of 34 71 Page of 34 71

Page of 35 71 Page of 35 71

Aler0ng13 with13 ELK13 ELK13 examines13 the13 log13 records13 in13 process13 to13 send13 alerts13 to13 third-shy‐party13 alerng13 event13 management13 and13 social13 media13 applicaons13 As13 shown13 previously13 in13 the13 ELK13 architecture13 the13 ELK13 strategy13 for13 alerng13 is13 to13 send13 alerts13 from13 the13 Logstash13 component13 directly13 Therefore13 log13 records13 and13 alerts13 are13 sent13 to13 both13 Elascsearch13 and13 third-shy‐party13 tools13 at13 the13 same13 instant13 using13 the13 output13 secon13 of13 the13 Logstash13 configuraon13 Any13 alerts13 are13 generated13 during13 the13 filter13 secon13 processing13 of13 the13 Logstash13 configuraon13

Create13 alerts13 in13 Logstash13 The13 Logstash13 filter13 secon13 is13 used13 to13 trigger13 alerts13 based13 on13 the13 log13 records13 being13 processed13 For13 example13 log13 records13 can13 be13 checked13 for13 errors13 warnings13 crashed13 or13 stopped13 messages13 and13 generate13 an13 alert13

Here13 is13 an13 example13 Logstash13 filter13 code13

13 if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

The13 specific13 tags13 are13 set13 against13 the13 log13 records13 to13 trigger13 the13 alert13 later13 in13 the13 configuraon13 in13 the13 output13 secon13

Control13 the13 number13 of13 generated13 alerts13 using13 the13 Logstash13 throcle13 plugin13 so13 that13 either13 a13 lot13 of13 instances13 in13 a13 fixed13 me13 period13 are13 required13 before13 an13 alert13 is13 sent13 or13 a13 specific13 number13 of13 instances13 triggers13 the13 alert13

Page of 36 71 Page of 36 71

Edit13 the13 plugin13 parameters13 to13 meet13 system13 requirements13

13 13 13 13 13 if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 10

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle13

after_count =gt 20

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 30

period =gt 1200

key =gt servicename_alarm_warning

Page of 37 71 Page of 37 71

add_tag =gt servicename_throttled

13 13 13 13 13 13 13 13 13

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 40

period =gt 1500

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

Enrich13 alerts13 with13 message13 detail13 Some13 of13 the13 third-shy‐party13 integraons13 benefit13 from13 having13 components13 of13 the13 raw13 log13 record13 sent13 in13 addion13 to13 the13 raw13 record13 or13 fields13 in13 the13 record13 edited13 transformed13 or13 translated13

Grok13 Logstash13 allows13 the13 log13 record13 to13 be13 broken13 up13 into13 logical13 secons13 and13 each13 secon13 stored13 in13 a13 new13 parameter13 using13 the13 grok13 parameter13 in13 the13 filter13 secon13 Here13 is13 an13 example13 grok13

13 grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

To13 build13 the13 grok13 statement13 Logstash13 offers13 an13 online13 tool13 at13 hcpsgrokdebugherokuappcom13

Translate13 If13 specific13 fields13 in13 the13 log13 record13 are13 coded13 (for13 example13 the13 Bluemix13 app13 name13 is13 oaen13 not13 sent13 in13 the13 syslog13 messages13 instead13 the13 GUID13 is13 sent13 which13 is13 not13 useful)13 then13 the13 filter13 secon13 translate13 plugin13 can13 be13 used13 to13 add13 new13 fields13 by13 holding13 the13 decoded13 versions13 of13 those13 values13

13 translate

field =gt data1

destination =gt data1

override =gt true

Page of 38 71 Page of 38 71

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

dc50f6e4-c052-436a-bf27-95ba392c4228 monverifyservicemanagement

0c677e82-5bfc-4e38-9155-e416249e3ee6 ms-catalog-postpyretic-pomiculture

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

However13 the13 translate13 plugin13 is13 not13 included13 with13 the13 Logstash13 installaon13 In13 the13 next13 secon13 I13 will13 13 describe13 how13 to13 add13 new13 plugins13 to13 the13 Logstash13 instance13

Add13 custom13 plugins13 to13 Logstash13 As13 an13 open13 source13 tool13 Logstash13 encourages13 users13 to13 develop13 content13 for13 all13 users13 to13 share13 Some13 of13 the13 plugins13 that13 were13 created13 by13 our13 community13 include13 a13 translaon13 plugin13 an13 output13 Slack13 plugin13 and13 an13 output13 syslog13 plugin13 Letrsquos13 look13 at13 how13 to13 add13 these13 plugins13 to13 your13 environment13

Filter13 translate13 plugin13 To13 add13 the13 translate13 plugin13

1 Download13 the13 plugin13

2 Copy13 the13 plugin13 file13 logstash-shy‐filter-shy‐translate-shy‐214gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-filter-translate 214

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Output13 Slack13 plugin13 To13 add13 the13 Slack13 plugin13

Page of 39 71 Page of 39 71

1 Download13 the13 plugin13 13

2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐slack-shy‐014gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-output-slack 014

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Output13 syslog13 plugin13 To13 add13 the13 syslog13 plugin13

1 Download13 the13 plugin13 13

2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐syslog-shy‐214gem13 to13 the13 ELK13 Virtual13 Server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-output-syslog 214

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Integrate13 Logstash13 with13 Slack13 To13 send13 generated13 alerts13 to13 Slack13 complete13 these13 steps13

1 Create13 a13 Slack13 integraon13 13

2 Use13 the13 Slack13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 Slack13 integraon13

113 Login13 to13 Slack13 and13 go13 to13 the13 Apps13 page13 13

Page of 40 71 Page of 40 71

13

313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13

13

A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13

13

Page of 42 71 Page of 42 71

The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13

13

The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13

13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

slack

url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5

channel =gt cloud-operations-mon

username =gt nick_cawood

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13

Page of 43 71 Page of 43 71

Page of 44 71 Page of 44 71

In13 Slack13 the13 alerts13 look13 like13 this13

13

Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13

1 Create13 a13 PagerDuty13 integraon13 13

2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13

1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13

13

2 Click13 Add13 New13 Service13

13

313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13

Page of 45 71 Page of 45 71

13

413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13

513 Click13 Add13 Service13

13

The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13

Page of 46 71 Page of 46 71

13

The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt be7eb1ea9c224018923e818d3e891d0b

incident_key =gt logstashhosttype

Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13

Page of 47 71 Page of 47 71

13

Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13

1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13

2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13

3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13

4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13

13 13 http

url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo

http_method =gt post

format =gt json

For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13

13 13 syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

Page of 48 71 Page of 48 71

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13

13

The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13

if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2

This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13

Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13

Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13

Logstash13 severity Shown13 in13 Omnibus13 as

alert Indeterminate

crical Warning

debug 7

emergency Clear

error Minor

informaonal 6

noce Crical

warning Major

Page of 49 71 Page of 49 71

13

Page of 50 71 Page of 50 71

Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13

input

tcp

port =gt 5000

type =gt syslog

filter

mutate

add_field =gt format =gt cf

grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

translate

field =gt data1

destination =gt data1

override =gt true

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

=== Check Alarm Criteria ===

Standard Key Words

if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

=== Alarm Exclusions ===

if [message] =~ (i)sometexthere

mutate remove_tag =gt servicename_alarm

Throttle alarms so we do not hit PD rate limits

Page of 52 71 Page of 52 71

if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 5

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle

after_count =gt 5

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 5

period =gt 1200

key =gt servicename_alarm_warning

add_tag =gt servicename_throttled

Page of 53 71 Page of 53 71

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

output

elasticsearch

hosts =gt localhost

stdout codec =gt rubydebug

if servicename_alarm_err in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

Page of 54 71 Page of 54 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 55 71 Page of 55 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

Page of 56 71 Page of 56 71

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert err message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_error in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

Page of 57 71 Page of 57 71

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 58 71 Page of 58 71

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 59 71 Page of 59 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert error message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

Page of 60 71 Page of 60 71

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

Page of 61 71 Page of 61 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 62 71 Page of 62 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt x

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert fatal message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

Page of 63 71 Page of 63 71

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

Page of 64 71 Page of 64 71

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 65 71 Page of 65 71

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt heavy_exclamation_mark

Page of 66 71 Page of 66 71

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert WARNING message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 67 71 Page of 67 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

Page of 68 71 Page of 68 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 69 71 Page of 69 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

Page of 70 71 Page of 70 71

Page of 71 71 Page of 71 71

Page 14: How to integrate the Elasticsearch Logstash Kibana (ELK ... · How to integrate the Elasticsearch Logstash Kibana (ELK) log analytics stack into IBM Bluemix Nick Cawood IBM Cloud

13

Transfer13 the13 files13 to13 the13 virtual13 server13 (using13 the13 scp13 method13 detailed13 above)13

Install13 Elas0csearch13 As13 a13 root13 user13 create13 a13 non-shy‐root13 user13 with13 the13 adduser13 command13

Create13 a13 directory13 to13 install13 Elascsearch13 into13 and13 untar13 the13 contents13 of13 the13 Elascsearch13 install13 file13 into13 that13 directory13

[elkelkforcase]$ cd home

[elkelkforcase]$ mkdir elasticsearch

[elkelkforcase]$ cp homeibmcloudelasticsearch-233targz homeelasticsearch

[elkelkforcase]$ cd elasticsearch

[elkelkforcase]$ gunzip elasticsearch-233targz

[elkelkforcase]$ untar elasticsearch-233tar

Elascsearch13 is13 now13 installed13

Start13 Elas0csearch13 To13 start13 Elascsearch13 go13 to13 the13 bin13 directory13 as13 the13 non-shy‐root13 user13 when13 you13 installed13 it13 and13 run13 the13 elascsearch13 executable13

[elkelkforcase]$ cd homeelasticsearchelasticsearch-233bin

[elkelkforcase]$ elasticsearch amp

[2] 22211

[elkelkforcase]$ [2016-07-05 133857730][INFO ][node] [Arsenal] version[233] pid[22211] build[218bdf12016-05-17T154004Z]

[2016-07-05 133857740][INFO ][node] [Arsenal] initializing

[2016-07-05 133858984][INFO ][plugins] [Arsenal] modules [reindex lang-expression lang-groovy] plugins [] sites []

[2016-07-05 133859146][INFO ][env] [Arsenal] using [1] data paths mounts [[ (rootfs)]] net usable_space [155gb] net total_space [199gb] spins [unknown] types [rootfs]

[2016-07-05 133859146][INFO ][env] [Arsenal] heap size [10156mb] compressed ordinary object pointers [true]

Page of 14 71 Page of 14 71

[2016-07-05 133859147][WARN ][env] [Arsenal] max file descriptors [4096] for elasticsearch process likely too low consider increasing to at least [65536]

[2016-07-05 133903526][INFO ][node] [Arsenal] initialized

[2016-07-05 133903526][INFO ][node] [Arsenal] starting

To13 test13 that13 Elascsearch13 is13 running13 correctly13 open13 the13 browser13 in13 the13 VNC13 client13 and13 run13 localhost920013

13

Install13 Kibana13 ELK13 recommends13 installing13 and13 running13 Kibana13 as13 root13 13

1 As13 root13 user13 create13 an13 installaon13 directory13 and13 untar13 the13 installaon13 file13

[rootelkforcase]$ cd home

[rootelkforcase]$ mkdir kibana

[rootelkforcase]$ cp homeibmcloudkibana-451-linux-x64targz homekibana

[rootelkforcase]$ cd kibana

[rootelkforcase]$ gunzip kibana-451-linux-x64targz

[rootelkforcase]$ untar kibana-451-linux-x64tar

2 Modify13 the13 manifestyml13 configuraon13 file13 Go13 to13 the13 kibana-shy‐451-shy‐linux-shy‐x6413 directory13

[rootelkforcase]$ cd homekibanakibana-451-linux-x64

3 Edit13 the13 manifestyml13 file13 so13 the13 contents13 are13

applicaons13

-shy‐ name13 kibana-shy‐in-shy‐bluemix13

memory13 128M13

host13 kibana-shy‐in-shy‐bluemix13

buildpack13 hcpsgithubcomcloudfoundry-shy‐communitynginx-shy‐buildpackgit13

Page of 15 71 Page of 15 71

4 Edit13 the13 configkibanayml13 file13 so13 the13 contents13 are13

Kibana is served by a back end server This controls which port to use

port 5601

The host to bind the server to

host 0000

If you are running kibana behind a proxy and want to mount it at a path

specify that path here The basePath cant end in a slash

serverbasePath

The maximum payload size in bytes on incoming server requests

servermaxPayloadBytes 1048576

The Elasticsearch instance to use for all your queries

elasticsearchurl httplocalhost9200

preserve_elasticsearch_host true will send the hostname specified in `elasticsearch` If you set it to false

then the host you use to connect to this Kibana instance will be sent

elasticsearchpreserveHost true

Kibana uses an index in Elasticsearch to store saved searches visualizations

and dashboards It will create a new index if it doesnt already exist

kibanaindex kibana

The default application to load

kibanadefaultAppId discover

If your Elasticsearch is protected with basic auth these are the user credentials

used by the Kibana server to perform maintenance on the kibana_index at startup Your

Kibana users will still need to authenticate with Elasticsearch (which is proxied through

the Kibana server)

Page of 16 71 Page of 16 71

elasticsearchusername user

elasticsearchpassword pass

SSL for outgoing requests from the Kibana Server to the browser (PEM formatted)

serversslcert pathtoyourservercrt

serversslkey pathtoyourserverkey

Optional setting to validate that your Elasticsearch backend uses the same key files (PEM formatted)

elasticsearchsslcert pathtoyourclientcrt

elasticsearchsslkey pathtoyourclientkey

If you need to provide a CA certificate for your Elasticsearch instance put

the path of the pem file here

elasticsearchsslca pathtoyourCApem

Set to false to have a complete disregard for the validity of the SSL

certificate

elasticsearchsslverify true

Time in milliseconds to wait for elasticsearch to respond to pings defaults to

request_timeout setting

elasticsearchpingTimeout 1500

Time in milliseconds to wait for responses from the back end or elasticsearch

This must be gt 0

elasticsearchrequestTimeout 30000

Time in milliseconds for Elasticsearch to wait for responses from shards

Set to 0 to disable

elasticsearchshardTimeout 0

Time in milliseconds to wait for Elasticsearch at Kibana startup before retrying

elasticsearchstartupTimeout 5000

elasticsearchstartupTimeout 30000

Page of 17 71 Page of 17 71

Set the path to where you would like the process id file to be created

pidfile varrunkibanapid

If you would like to send the log output to a file you can set the path below

loggingdest stdout

Set this to true to suppress all logging output

loggingsilent false

Set this to true to suppress all logging output except for error messages

loggingquiet false

Set this to true to log all events including system usage information and all requests

loggingverbose false

Kibana13 is13 now13 installed13

Start13 Kibana13 To13 start13 Kibana13 go13 to13 the13 bin13 directory13 and13 run13 the13 kibana13 executable13

[rootelkforcase]$

[rootelkforcase]$ cd homekibanakibana-451-linux-x64bin

[rootelkforcase]$ kibana amp

[1] 22283

[rootelkforcase] log [095900542] [info][status][pluginkibana] Status changed from uninitialized to green - Ready

log [095900600] [info][status][pluginelasticsearch] Status changed from uninitialized to yellow - Waiting for Elasticsearch

log [095900689] [info][status][pluginkbn_vislib_vis_types] Status changed from uninitialized to green - Ready

log [095900717] [info][status][pluginmarkdown_vis] Status changed from uninitialized to green - Ready

log [095900721] [info][status][pluginmetric_vis] Status changed from uninitialized to green - Ready

log [095900814] [info][status][pluginspyModes] Status changed from uninitialized to green - Ready

log [095900934] [info][status][pluginstatusPage] Status changed from uninitialized to green - Ready

Page of 18 71 Page of 18 71

log [095900941] [info][status][plugintable_vis] Status changed from uninitialized to green - Ready

log [095901067] [info][listening] Server running at http00005601

log [095906083] [info][status][pluginelasticsearch] Status changed from yellow to yellow - No existing Kibana index found

log [095908942] [info][status][pluginelasticsearch] Status changed from yellow to green - Kibana index ready

Kibana13 is13 now13 started13

Broadcast13 Bluemix13 app13 log13 records13 Bluemix13 apps13 (which13 are13 shown13 in13 the13 US13 South13 Bluemix13 region13 of13 the13 System13 Architecture13 diagram)13 will13 produce13 logs13 in13 a13 syslog13 rfc542413 format13 using13 a13 Bluemix13 tool13 called13 Loggregator13

For13 Logstash13 to13 receive13 these13 logs13 each13 Bluemix13 app13 needs13 to13 broadcast13 these13 logs13 To13 do13 that13 run13 the13 following13 commands13 on13 each13 Bluemix13 app13 using13 the13 public13 IP13 address13 of13 the13 ELK13 virtual13 server13 the13 ELK13 tag13 to13 trace13 the13 process13 and13 insert13 the13 name13 of13 each13 app13 in13 turn13

cf cups elk -l syslogltpublicIPaddress5000 cf bind-service ltmyappgt elk

cf restage ltmyappgt

Install13 Logstash13 To13 install13 Logstash13 as13 root13 user13 create13 a13 directory13 for13 the13 installaon13 and13 untar13 the13 installaon13 file13 there13

13 13 [rootelkforcase]$ cd home

[rootelkforcase]$ mkdir logstash

[rootelkforcase]$ cp homeibmcloudlogstash-233targz homelogstash

[rootelkforcase]$ cd logstash

[rootelkforcase]$ gunzip logstash-233targz

[rootelkforcase]$ untar logstash-233tar

Write13 Logstash13 configura0on13 Logstash13 requires13 a13 configuraon13 file13 to13 be13 wricen13 to13 receive13 transform13 and13 output13 the13 log13 records13 to13 Elascsearch13 This13 secon13 explains13 the13 basic13 requirements13 of13 processing13 Bluemix13 app13 log13 records13 into13 Elascsearch13 (more13 complex13 configuraons13 are13 detailed13 later13 in13 this13 document13 for13 the13 integraons13 with13 third-shy‐party13 tools)13

Create13 a13 conf13 file13 as13 root13 user13

[rootelkforcase]$ cd homelogstashlogstash-233

[rootelkforcase]$ mkdir config

Page of 19 71 Page of 19 71

[rootelkforcase]$ cd config

[rootelkforcase]$ touch loggregatorconf

Edit13 the13 loggregator13 file13 with13 the13 following13 contents13

13 Input13 secon13 defines13 which13 log13 records13 to13 receive13

13 port13 and13 type13 reflect13 method13 selected13 for13 Cloud13 Foundry13 to13 broadcast13 the13 records13

input

tcp

port =gt 5000

type =gt syslog

13 Filter13 secon13 ndash13 transforms13 the13 data13

filter

Add a field to identify the records as Cloud Foundry based

mutate

add_field =gt format =gt cf

13 Output13 secon13 ndash13 sends13 the13 data13 to13 its13 desnaon(s)13

output

13 Use13 the13 elascsearch13 plugin13 to13 send13 the13 data13 to13 the13 elascsearch13 instance13

13 elasticsearch

hosts =gt localhost

stdout codec =gt rubydebug

13 End13 of13 Config13

Save13 the13 file13 Verify13 the13 format13 of13 the13 file13 is13 acceptable13 by13 running13 the13 following13 commands13

[rootelkforcase]$ cd homelogstashlogstash-233

[rootelkforcase]$ binlogstash agent --configtest -f configloggregatorconf

Page of 20 71 Page of 20 71

Configuraon13 is13 OK13

Start13 Logstash13 Aaer13 you13 have13 wricen13 and13 verified13 the13 Logstash13 configuraon13 you13 can13 start13 Logstash13 to13 process13 records13 from13 the13 Bluemix13 apps13 (via13 Cloud13 Foundry)13 Run13 the13 following13 as13 root13 user13

[rootelkforcase]$ cd homelogstashlogstash-233

[rootelkforcase]$ binlogstash agent -f configloggregatorconf

Logstash13 is13 now13 started13

Monitor13 Logstash13 processing13 Once13 Logstash13 is13 running13 and13 log13 records13 are13 being13 produced13 by13 the13 Bluemix13 apps13 you13 can13 monitor13 progress13 in13 the13 same13 window13 that13 was13 used13 to13 start13 Logstash13 Output13 is13 similar13 to13 the13 following13

13

The13 log13 records13 being13 broadcast13 from13 the13 Bluemix13 apps13 are13 now13 sent13 to13 Elascsearch13 from13 the13 Logstash13 output13 configuraon13 Records13 are13 sent13 in13 real13 me13 If13 Logstash13 is13 not13 running13 while13 log13 records13 are13 being13 produced13 then13 these13 records13 will13 not13 be13 processed13 by13 Logstash13 and13 will13 not13 be13 stored13 in13 Elascsearch13

Page of 21 71 Page of 21 71

Use13 ELK13 The13 log13 records13 from13 the13 Bluemix13 apps13 are13 now13 being13 stored13 and13 indexed13 by13 the13 Elascsearch13 component13 This13 data13 can13 be13 accessed13 via13 the13 Kibana13 applicaon13

Kibana13 Discover13 page13 Using13 a13 web13 browser13 either13 directly13 from13 a13 client13 or13 using13 the13 VNC13 client13 access13 Kibana13 using13 the13 publicIPaddress13 of13 the13 ELK13 virtual13 server13

hcpltpublicIPaddressgt560113

No13 userpassword13 credenals13 are13 required13

Kibana13 defaults13 to13 a13 search13 of13 all13 records13 ()13 in13 the13 last13 1513 minutes13 and13 returns13 the13 results13

13

More13 complex13 searches13 are13 possible13 by13 changing13 the13 me13 filter13 and13 supplying13 search13 queries13

Change13 the13 0me13 filter13 To13 change13 the13 me13 ranges13 click13 the13 clock13 symbol13 in13 the13 top-shy‐right13 of13 the13 Kibana13 Discover13 page13

13

You13 will13 see13 the13 following13 opons13

13

Page of 22 71 Page of 22 71

A13 variety13 of13 quick13 me13 ranges13 can13 be13 selected13 or13 you13 can13 switch13 to13 relave13 or13 absolute13 me13 ranges13

Quick13 me13 ranges13 allow13 switching13 between13 last13 X13 minutes13 to13 last13 Y13 hours13 or13 Z13 days13 etc13

Relave13 me13 ranges13 allow13 more13 specific13 versions13 of13 the13 Quick13 opons13 from13 specific13 minuteshoursdays13 to13 now13

13

Absolute13 me13 ranges13 specify13 the13 start13 and13 end13 mes13 for13 the13 searches13

13

Custom13 search13 queries13 You13 can13 use13 the13 text13 window13 of13 the13 Discover13 page13 to13 search13 for13 matches13 within13 the13 log13 records13 You13 can13 use13 Apache13 Lucene13 or13 Elascsearch13 query13 DSL13 syntax13

13

Page of 23 71 Page of 23 71

Saved13 searches13 You13 can13 save13 any13 custom13 search13 syntax13 and13 choice13 of13 me13 range13 for13 future13 ease13 of13 use13 Any13 relave13 or13 absolute13 me13 frame13 is13 saved13 as13 part13 of13 the13 search13 13

To13 save13 a13 search13

1 Set13 up13 and13 run13 the13 required13 syntax13 and13 me13 frame13

13

2 Click13 the13 Save13 Search13 icon13

13

3 Type13 in13 a13 name13 for13 the13 saved13 search13

Page of 24 71 Page of 24 71

13

7 To13 delete13 or13 update13 saved13 searches13 click13 13 manage13 saved13 searches13

13

Use13 the13 edit13 icons13 to13 make13 any13 changes13

Page of 26 71 Page of 26 71

13

Build13 simple13 Kibana13 histograms13 Kibana13 allows13 you13 to13 build13 charts13 and13 dashboards13 based13 on13 the13 search13 capability13 offered13 by13 the13 Discover13 page13 Saved13 searches13 can13 be13 used13 as13 the13 basis13 for13 charts13 and13 dashboards13 built13 using13 those13 charts13 To13 build13 a13 simple13 histogram13

1 Select13 the13 Visualize13 page13

13

2 Select13 Ver0cal13 Bar13 Chart13

13

3 To13 use13 a13 saved13 search13 click13 From13 a13 saved13 search13

Page of 27 71 Page of 27 71

13

4 Select13 a13 previously13 saved13 search13

13

5 Select13 to13 edit13 the13 x-shy‐axis

6 Edit13 the13 fields13 with13 informaon13 about13 the13 aggregaon13 field13 interval13 and13 custom13 label13

Page of 28 71 Page of 28 71

13

Page of 29 71 Page of 29 71

7 Click13 the13 green13 arrow13 The13 histogram13 is13 created13 and13 run13 based13 on13 the13 saved13 search

13

8 To13 save13 the13 histogram13 click13 the13 Save13 Visualiza0on13 icon13 and13 add13 a13 tle13

13

13

9 Click13 Save13

10 To13 view13 the13 visualisaon13 click13 the13 Dashboard13 page13

Page of 30 71 Page of 30 71

13

11 Click13 the13 Add13 Visualiza0on13 icon13

13

12 Select13 the13 visiualizaon13 that13 you13 just13 created13

13

Page of 31 71 Page of 31 71

13

13 To13 save13 the13 dashboard13 click13 the13 Save13 Dashboard13 icon13

Page of 32 71 Page of 32 71

13

14 Add13 a13 tle13 and13 select13 whether13 to13 store13 the13 selected13 me13 frame13 with13 the13 dashboard13

13

15 Click13 Save13

16 You13 can13 edit13 the13 shape13 and13 number13 of13 charts13 and13 save13 the13 dashboard13

17 To13 re-shy‐run13 the13 dashboard13 select13 the13 Dashboard13 page13 and13 click13 the13 Load13 Saved13 Dashboard13 icon13

Page of 33 71 Page of 33 71

13

18 Select13 the13 saved13 dashboard13 from13 the13 list13

13

19 The13 dashboard13 runs13

13

Page of 34 71 Page of 34 71

Page of 35 71 Page of 35 71

Aler0ng13 with13 ELK13 ELK13 examines13 the13 log13 records13 in13 process13 to13 send13 alerts13 to13 third-shy‐party13 alerng13 event13 management13 and13 social13 media13 applicaons13 As13 shown13 previously13 in13 the13 ELK13 architecture13 the13 ELK13 strategy13 for13 alerng13 is13 to13 send13 alerts13 from13 the13 Logstash13 component13 directly13 Therefore13 log13 records13 and13 alerts13 are13 sent13 to13 both13 Elascsearch13 and13 third-shy‐party13 tools13 at13 the13 same13 instant13 using13 the13 output13 secon13 of13 the13 Logstash13 configuraon13 Any13 alerts13 are13 generated13 during13 the13 filter13 secon13 processing13 of13 the13 Logstash13 configuraon13

Create13 alerts13 in13 Logstash13 The13 Logstash13 filter13 secon13 is13 used13 to13 trigger13 alerts13 based13 on13 the13 log13 records13 being13 processed13 For13 example13 log13 records13 can13 be13 checked13 for13 errors13 warnings13 crashed13 or13 stopped13 messages13 and13 generate13 an13 alert13

Here13 is13 an13 example13 Logstash13 filter13 code13

13 if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

The13 specific13 tags13 are13 set13 against13 the13 log13 records13 to13 trigger13 the13 alert13 later13 in13 the13 configuraon13 in13 the13 output13 secon13

Control13 the13 number13 of13 generated13 alerts13 using13 the13 Logstash13 throcle13 plugin13 so13 that13 either13 a13 lot13 of13 instances13 in13 a13 fixed13 me13 period13 are13 required13 before13 an13 alert13 is13 sent13 or13 a13 specific13 number13 of13 instances13 triggers13 the13 alert13

Page of 36 71 Page of 36 71

Edit13 the13 plugin13 parameters13 to13 meet13 system13 requirements13

13 13 13 13 13 if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 10

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle13

after_count =gt 20

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 30

period =gt 1200

key =gt servicename_alarm_warning

Page of 37 71 Page of 37 71

add_tag =gt servicename_throttled

13 13 13 13 13 13 13 13 13

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 40

period =gt 1500

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

Enrich13 alerts13 with13 message13 detail13 Some13 of13 the13 third-shy‐party13 integraons13 benefit13 from13 having13 components13 of13 the13 raw13 log13 record13 sent13 in13 addion13 to13 the13 raw13 record13 or13 fields13 in13 the13 record13 edited13 transformed13 or13 translated13

Grok13 Logstash13 allows13 the13 log13 record13 to13 be13 broken13 up13 into13 logical13 secons13 and13 each13 secon13 stored13 in13 a13 new13 parameter13 using13 the13 grok13 parameter13 in13 the13 filter13 secon13 Here13 is13 an13 example13 grok13

13 grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

To13 build13 the13 grok13 statement13 Logstash13 offers13 an13 online13 tool13 at13 hcpsgrokdebugherokuappcom13

Translate13 If13 specific13 fields13 in13 the13 log13 record13 are13 coded13 (for13 example13 the13 Bluemix13 app13 name13 is13 oaen13 not13 sent13 in13 the13 syslog13 messages13 instead13 the13 GUID13 is13 sent13 which13 is13 not13 useful)13 then13 the13 filter13 secon13 translate13 plugin13 can13 be13 used13 to13 add13 new13 fields13 by13 holding13 the13 decoded13 versions13 of13 those13 values13

13 translate

field =gt data1

destination =gt data1

override =gt true

Page of 38 71 Page of 38 71

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

dc50f6e4-c052-436a-bf27-95ba392c4228 monverifyservicemanagement

0c677e82-5bfc-4e38-9155-e416249e3ee6 ms-catalog-postpyretic-pomiculture

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

However13 the13 translate13 plugin13 is13 not13 included13 with13 the13 Logstash13 installaon13 In13 the13 next13 secon13 I13 will13 13 describe13 how13 to13 add13 new13 plugins13 to13 the13 Logstash13 instance13

Add13 custom13 plugins13 to13 Logstash13 As13 an13 open13 source13 tool13 Logstash13 encourages13 users13 to13 develop13 content13 for13 all13 users13 to13 share13 Some13 of13 the13 plugins13 that13 were13 created13 by13 our13 community13 include13 a13 translaon13 plugin13 an13 output13 Slack13 plugin13 and13 an13 output13 syslog13 plugin13 Letrsquos13 look13 at13 how13 to13 add13 these13 plugins13 to13 your13 environment13

Filter13 translate13 plugin13 To13 add13 the13 translate13 plugin13

1 Download13 the13 plugin13

2 Copy13 the13 plugin13 file13 logstash-shy‐filter-shy‐translate-shy‐214gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-filter-translate 214

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Output13 Slack13 plugin13 To13 add13 the13 Slack13 plugin13

Page of 39 71 Page of 39 71

1 Download13 the13 plugin13 13

2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐slack-shy‐014gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-output-slack 014

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Output13 syslog13 plugin13 To13 add13 the13 syslog13 plugin13

1 Download13 the13 plugin13 13

2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐syslog-shy‐214gem13 to13 the13 ELK13 Virtual13 Server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-output-syslog 214

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Integrate13 Logstash13 with13 Slack13 To13 send13 generated13 alerts13 to13 Slack13 complete13 these13 steps13

1 Create13 a13 Slack13 integraon13 13

2 Use13 the13 Slack13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 Slack13 integraon13

113 Login13 to13 Slack13 and13 go13 to13 the13 Apps13 page13 13

Page of 40 71 Page of 40 71

13

313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13

13

A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13

13

Page of 42 71 Page of 42 71

The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13

13

The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13

13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

slack

url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5

channel =gt cloud-operations-mon

username =gt nick_cawood

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13

Page of 43 71 Page of 43 71

Page of 44 71 Page of 44 71

In13 Slack13 the13 alerts13 look13 like13 this13

13

Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13

1 Create13 a13 PagerDuty13 integraon13 13

2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13

1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13

13

2 Click13 Add13 New13 Service13

13

313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13

Page of 45 71 Page of 45 71

13

413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13

513 Click13 Add13 Service13

13

The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13

Page of 46 71 Page of 46 71

13

The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt be7eb1ea9c224018923e818d3e891d0b

incident_key =gt logstashhosttype

Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13

Page of 47 71 Page of 47 71

13

Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13

1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13

2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13

3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13

4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13

13 13 http

url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo

http_method =gt post

format =gt json

For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13

13 13 syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

Page of 48 71 Page of 48 71

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13

13

The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13

if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2

This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13

Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13

Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13

Logstash13 severity Shown13 in13 Omnibus13 as

alert Indeterminate

crical Warning

debug 7

emergency Clear

error Minor

informaonal 6

noce Crical

warning Major

Page of 49 71 Page of 49 71

13

Page of 50 71 Page of 50 71

Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13

input

tcp

port =gt 5000

type =gt syslog

filter

mutate

add_field =gt format =gt cf

grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

translate

field =gt data1

destination =gt data1

override =gt true

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

=== Check Alarm Criteria ===

Standard Key Words

if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

=== Alarm Exclusions ===

if [message] =~ (i)sometexthere

mutate remove_tag =gt servicename_alarm

Throttle alarms so we do not hit PD rate limits

Page of 52 71 Page of 52 71

if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 5

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle

after_count =gt 5

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 5

period =gt 1200

key =gt servicename_alarm_warning

add_tag =gt servicename_throttled

Page of 53 71 Page of 53 71

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

output

elasticsearch

hosts =gt localhost

stdout codec =gt rubydebug

if servicename_alarm_err in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

Page of 54 71 Page of 54 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 55 71 Page of 55 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

Page of 56 71 Page of 56 71

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert err message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_error in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

Page of 57 71 Page of 57 71

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 58 71 Page of 58 71

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 59 71 Page of 59 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert error message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

Page of 60 71 Page of 60 71

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

Page of 61 71 Page of 61 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 62 71 Page of 62 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt x

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert fatal message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

Page of 63 71 Page of 63 71

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

Page of 64 71 Page of 64 71

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 65 71 Page of 65 71

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt heavy_exclamation_mark

Page of 66 71 Page of 66 71

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert WARNING message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 67 71 Page of 67 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

Page of 68 71 Page of 68 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 69 71 Page of 69 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

Page of 70 71 Page of 70 71

Page of 71 71 Page of 71 71

Page 15: How to integrate the Elasticsearch Logstash Kibana (ELK ... · How to integrate the Elasticsearch Logstash Kibana (ELK) log analytics stack into IBM Bluemix Nick Cawood IBM Cloud

[2016-07-05 133859147][WARN ][env] [Arsenal] max file descriptors [4096] for elasticsearch process likely too low consider increasing to at least [65536]

[2016-07-05 133903526][INFO ][node] [Arsenal] initialized

[2016-07-05 133903526][INFO ][node] [Arsenal] starting

To13 test13 that13 Elascsearch13 is13 running13 correctly13 open13 the13 browser13 in13 the13 VNC13 client13 and13 run13 localhost920013

13

Install13 Kibana13 ELK13 recommends13 installing13 and13 running13 Kibana13 as13 root13 13

1 As13 root13 user13 create13 an13 installaon13 directory13 and13 untar13 the13 installaon13 file13

[rootelkforcase]$ cd home

[rootelkforcase]$ mkdir kibana

[rootelkforcase]$ cp homeibmcloudkibana-451-linux-x64targz homekibana

[rootelkforcase]$ cd kibana

[rootelkforcase]$ gunzip kibana-451-linux-x64targz

[rootelkforcase]$ untar kibana-451-linux-x64tar

2 Modify13 the13 manifestyml13 configuraon13 file13 Go13 to13 the13 kibana-shy‐451-shy‐linux-shy‐x6413 directory13

[rootelkforcase]$ cd homekibanakibana-451-linux-x64

3 Edit13 the13 manifestyml13 file13 so13 the13 contents13 are13

applicaons13

-shy‐ name13 kibana-shy‐in-shy‐bluemix13

memory13 128M13

host13 kibana-shy‐in-shy‐bluemix13

buildpack13 hcpsgithubcomcloudfoundry-shy‐communitynginx-shy‐buildpackgit13

Page of 15 71 Page of 15 71

4 Edit13 the13 configkibanayml13 file13 so13 the13 contents13 are13

Kibana is served by a back end server This controls which port to use

port 5601

The host to bind the server to

host 0000

If you are running kibana behind a proxy and want to mount it at a path

specify that path here The basePath cant end in a slash

serverbasePath

The maximum payload size in bytes on incoming server requests

servermaxPayloadBytes 1048576

The Elasticsearch instance to use for all your queries

elasticsearchurl httplocalhost9200

preserve_elasticsearch_host true will send the hostname specified in `elasticsearch` If you set it to false

then the host you use to connect to this Kibana instance will be sent

elasticsearchpreserveHost true

Kibana uses an index in Elasticsearch to store saved searches visualizations

and dashboards It will create a new index if it doesnt already exist

kibanaindex kibana

The default application to load

kibanadefaultAppId discover

If your Elasticsearch is protected with basic auth these are the user credentials

used by the Kibana server to perform maintenance on the kibana_index at startup Your

Kibana users will still need to authenticate with Elasticsearch (which is proxied through

the Kibana server)

Page of 16 71 Page of 16 71

elasticsearchusername user

elasticsearchpassword pass

SSL for outgoing requests from the Kibana Server to the browser (PEM formatted)

serversslcert pathtoyourservercrt

serversslkey pathtoyourserverkey

Optional setting to validate that your Elasticsearch backend uses the same key files (PEM formatted)

elasticsearchsslcert pathtoyourclientcrt

elasticsearchsslkey pathtoyourclientkey

If you need to provide a CA certificate for your Elasticsearch instance put

the path of the pem file here

elasticsearchsslca pathtoyourCApem

Set to false to have a complete disregard for the validity of the SSL

certificate

elasticsearchsslverify true

Time in milliseconds to wait for elasticsearch to respond to pings defaults to

request_timeout setting

elasticsearchpingTimeout 1500

Time in milliseconds to wait for responses from the back end or elasticsearch

This must be gt 0

elasticsearchrequestTimeout 30000

Time in milliseconds for Elasticsearch to wait for responses from shards

Set to 0 to disable

elasticsearchshardTimeout 0

Time in milliseconds to wait for Elasticsearch at Kibana startup before retrying

elasticsearchstartupTimeout 5000

elasticsearchstartupTimeout 30000

Page of 17 71 Page of 17 71

Set the path to where you would like the process id file to be created

pidfile varrunkibanapid

If you would like to send the log output to a file you can set the path below

loggingdest stdout

Set this to true to suppress all logging output

loggingsilent false

Set this to true to suppress all logging output except for error messages

loggingquiet false

Set this to true to log all events including system usage information and all requests

loggingverbose false

Kibana13 is13 now13 installed13

Start13 Kibana13 To13 start13 Kibana13 go13 to13 the13 bin13 directory13 and13 run13 the13 kibana13 executable13

[rootelkforcase]$

[rootelkforcase]$ cd homekibanakibana-451-linux-x64bin

[rootelkforcase]$ kibana amp

[1] 22283

[rootelkforcase] log [095900542] [info][status][pluginkibana] Status changed from uninitialized to green - Ready

log [095900600] [info][status][pluginelasticsearch] Status changed from uninitialized to yellow - Waiting for Elasticsearch

log [095900689] [info][status][pluginkbn_vislib_vis_types] Status changed from uninitialized to green - Ready

log [095900717] [info][status][pluginmarkdown_vis] Status changed from uninitialized to green - Ready

log [095900721] [info][status][pluginmetric_vis] Status changed from uninitialized to green - Ready

log [095900814] [info][status][pluginspyModes] Status changed from uninitialized to green - Ready

log [095900934] [info][status][pluginstatusPage] Status changed from uninitialized to green - Ready

Page of 18 71 Page of 18 71

log [095900941] [info][status][plugintable_vis] Status changed from uninitialized to green - Ready

log [095901067] [info][listening] Server running at http00005601

log [095906083] [info][status][pluginelasticsearch] Status changed from yellow to yellow - No existing Kibana index found

log [095908942] [info][status][pluginelasticsearch] Status changed from yellow to green - Kibana index ready

Kibana13 is13 now13 started13

Broadcast13 Bluemix13 app13 log13 records13 Bluemix13 apps13 (which13 are13 shown13 in13 the13 US13 South13 Bluemix13 region13 of13 the13 System13 Architecture13 diagram)13 will13 produce13 logs13 in13 a13 syslog13 rfc542413 format13 using13 a13 Bluemix13 tool13 called13 Loggregator13

For13 Logstash13 to13 receive13 these13 logs13 each13 Bluemix13 app13 needs13 to13 broadcast13 these13 logs13 To13 do13 that13 run13 the13 following13 commands13 on13 each13 Bluemix13 app13 using13 the13 public13 IP13 address13 of13 the13 ELK13 virtual13 server13 the13 ELK13 tag13 to13 trace13 the13 process13 and13 insert13 the13 name13 of13 each13 app13 in13 turn13

cf cups elk -l syslogltpublicIPaddress5000 cf bind-service ltmyappgt elk

cf restage ltmyappgt

Install13 Logstash13 To13 install13 Logstash13 as13 root13 user13 create13 a13 directory13 for13 the13 installaon13 and13 untar13 the13 installaon13 file13 there13

13 13 [rootelkforcase]$ cd home

[rootelkforcase]$ mkdir logstash

[rootelkforcase]$ cp homeibmcloudlogstash-233targz homelogstash

[rootelkforcase]$ cd logstash

[rootelkforcase]$ gunzip logstash-233targz

[rootelkforcase]$ untar logstash-233tar

Write13 Logstash13 configura0on13 Logstash13 requires13 a13 configuraon13 file13 to13 be13 wricen13 to13 receive13 transform13 and13 output13 the13 log13 records13 to13 Elascsearch13 This13 secon13 explains13 the13 basic13 requirements13 of13 processing13 Bluemix13 app13 log13 records13 into13 Elascsearch13 (more13 complex13 configuraons13 are13 detailed13 later13 in13 this13 document13 for13 the13 integraons13 with13 third-shy‐party13 tools)13

Create13 a13 conf13 file13 as13 root13 user13

[rootelkforcase]$ cd homelogstashlogstash-233

[rootelkforcase]$ mkdir config

Page of 19 71 Page of 19 71

[rootelkforcase]$ cd config

[rootelkforcase]$ touch loggregatorconf

Edit13 the13 loggregator13 file13 with13 the13 following13 contents13

13 Input13 secon13 defines13 which13 log13 records13 to13 receive13

13 port13 and13 type13 reflect13 method13 selected13 for13 Cloud13 Foundry13 to13 broadcast13 the13 records13

input

tcp

port =gt 5000

type =gt syslog

13 Filter13 secon13 ndash13 transforms13 the13 data13

filter

Add a field to identify the records as Cloud Foundry based

mutate

add_field =gt format =gt cf

13 Output13 secon13 ndash13 sends13 the13 data13 to13 its13 desnaon(s)13

output

13 Use13 the13 elascsearch13 plugin13 to13 send13 the13 data13 to13 the13 elascsearch13 instance13

13 elasticsearch

hosts =gt localhost

stdout codec =gt rubydebug

13 End13 of13 Config13

Save13 the13 file13 Verify13 the13 format13 of13 the13 file13 is13 acceptable13 by13 running13 the13 following13 commands13

[rootelkforcase]$ cd homelogstashlogstash-233

[rootelkforcase]$ binlogstash agent --configtest -f configloggregatorconf

Page of 20 71 Page of 20 71

Configuraon13 is13 OK13

Start13 Logstash13 Aaer13 you13 have13 wricen13 and13 verified13 the13 Logstash13 configuraon13 you13 can13 start13 Logstash13 to13 process13 records13 from13 the13 Bluemix13 apps13 (via13 Cloud13 Foundry)13 Run13 the13 following13 as13 root13 user13

[rootelkforcase]$ cd homelogstashlogstash-233

[rootelkforcase]$ binlogstash agent -f configloggregatorconf

Logstash13 is13 now13 started13

Monitor13 Logstash13 processing13 Once13 Logstash13 is13 running13 and13 log13 records13 are13 being13 produced13 by13 the13 Bluemix13 apps13 you13 can13 monitor13 progress13 in13 the13 same13 window13 that13 was13 used13 to13 start13 Logstash13 Output13 is13 similar13 to13 the13 following13

13

The13 log13 records13 being13 broadcast13 from13 the13 Bluemix13 apps13 are13 now13 sent13 to13 Elascsearch13 from13 the13 Logstash13 output13 configuraon13 Records13 are13 sent13 in13 real13 me13 If13 Logstash13 is13 not13 running13 while13 log13 records13 are13 being13 produced13 then13 these13 records13 will13 not13 be13 processed13 by13 Logstash13 and13 will13 not13 be13 stored13 in13 Elascsearch13

Page of 21 71 Page of 21 71

Use13 ELK13 The13 log13 records13 from13 the13 Bluemix13 apps13 are13 now13 being13 stored13 and13 indexed13 by13 the13 Elascsearch13 component13 This13 data13 can13 be13 accessed13 via13 the13 Kibana13 applicaon13

Kibana13 Discover13 page13 Using13 a13 web13 browser13 either13 directly13 from13 a13 client13 or13 using13 the13 VNC13 client13 access13 Kibana13 using13 the13 publicIPaddress13 of13 the13 ELK13 virtual13 server13

hcpltpublicIPaddressgt560113

No13 userpassword13 credenals13 are13 required13

Kibana13 defaults13 to13 a13 search13 of13 all13 records13 ()13 in13 the13 last13 1513 minutes13 and13 returns13 the13 results13

13

More13 complex13 searches13 are13 possible13 by13 changing13 the13 me13 filter13 and13 supplying13 search13 queries13

Change13 the13 0me13 filter13 To13 change13 the13 me13 ranges13 click13 the13 clock13 symbol13 in13 the13 top-shy‐right13 of13 the13 Kibana13 Discover13 page13

13

You13 will13 see13 the13 following13 opons13

13

Page of 22 71 Page of 22 71

A13 variety13 of13 quick13 me13 ranges13 can13 be13 selected13 or13 you13 can13 switch13 to13 relave13 or13 absolute13 me13 ranges13

Quick13 me13 ranges13 allow13 switching13 between13 last13 X13 minutes13 to13 last13 Y13 hours13 or13 Z13 days13 etc13

Relave13 me13 ranges13 allow13 more13 specific13 versions13 of13 the13 Quick13 opons13 from13 specific13 minuteshoursdays13 to13 now13

13

Absolute13 me13 ranges13 specify13 the13 start13 and13 end13 mes13 for13 the13 searches13

13

Custom13 search13 queries13 You13 can13 use13 the13 text13 window13 of13 the13 Discover13 page13 to13 search13 for13 matches13 within13 the13 log13 records13 You13 can13 use13 Apache13 Lucene13 or13 Elascsearch13 query13 DSL13 syntax13

13

Page of 23 71 Page of 23 71

Saved13 searches13 You13 can13 save13 any13 custom13 search13 syntax13 and13 choice13 of13 me13 range13 for13 future13 ease13 of13 use13 Any13 relave13 or13 absolute13 me13 frame13 is13 saved13 as13 part13 of13 the13 search13 13

To13 save13 a13 search13

1 Set13 up13 and13 run13 the13 required13 syntax13 and13 me13 frame13

13

2 Click13 the13 Save13 Search13 icon13

13

3 Type13 in13 a13 name13 for13 the13 saved13 search13

Page of 24 71 Page of 24 71

13

7 To13 delete13 or13 update13 saved13 searches13 click13 13 manage13 saved13 searches13

13

Use13 the13 edit13 icons13 to13 make13 any13 changes13

Page of 26 71 Page of 26 71

13

Build13 simple13 Kibana13 histograms13 Kibana13 allows13 you13 to13 build13 charts13 and13 dashboards13 based13 on13 the13 search13 capability13 offered13 by13 the13 Discover13 page13 Saved13 searches13 can13 be13 used13 as13 the13 basis13 for13 charts13 and13 dashboards13 built13 using13 those13 charts13 To13 build13 a13 simple13 histogram13

1 Select13 the13 Visualize13 page13

13

2 Select13 Ver0cal13 Bar13 Chart13

13

3 To13 use13 a13 saved13 search13 click13 From13 a13 saved13 search13

Page of 27 71 Page of 27 71

13

4 Select13 a13 previously13 saved13 search13

13

5 Select13 to13 edit13 the13 x-shy‐axis

6 Edit13 the13 fields13 with13 informaon13 about13 the13 aggregaon13 field13 interval13 and13 custom13 label13

Page of 28 71 Page of 28 71

13

Page of 29 71 Page of 29 71

7 Click13 the13 green13 arrow13 The13 histogram13 is13 created13 and13 run13 based13 on13 the13 saved13 search

13

8 To13 save13 the13 histogram13 click13 the13 Save13 Visualiza0on13 icon13 and13 add13 a13 tle13

13

13

9 Click13 Save13

10 To13 view13 the13 visualisaon13 click13 the13 Dashboard13 page13

Page of 30 71 Page of 30 71

13

11 Click13 the13 Add13 Visualiza0on13 icon13

13

12 Select13 the13 visiualizaon13 that13 you13 just13 created13

13

Page of 31 71 Page of 31 71

13

13 To13 save13 the13 dashboard13 click13 the13 Save13 Dashboard13 icon13

Page of 32 71 Page of 32 71

13

14 Add13 a13 tle13 and13 select13 whether13 to13 store13 the13 selected13 me13 frame13 with13 the13 dashboard13

13

15 Click13 Save13

16 You13 can13 edit13 the13 shape13 and13 number13 of13 charts13 and13 save13 the13 dashboard13

17 To13 re-shy‐run13 the13 dashboard13 select13 the13 Dashboard13 page13 and13 click13 the13 Load13 Saved13 Dashboard13 icon13

Page of 33 71 Page of 33 71

13

18 Select13 the13 saved13 dashboard13 from13 the13 list13

13

19 The13 dashboard13 runs13

13

Page of 34 71 Page of 34 71

Page of 35 71 Page of 35 71

Aler0ng13 with13 ELK13 ELK13 examines13 the13 log13 records13 in13 process13 to13 send13 alerts13 to13 third-shy‐party13 alerng13 event13 management13 and13 social13 media13 applicaons13 As13 shown13 previously13 in13 the13 ELK13 architecture13 the13 ELK13 strategy13 for13 alerng13 is13 to13 send13 alerts13 from13 the13 Logstash13 component13 directly13 Therefore13 log13 records13 and13 alerts13 are13 sent13 to13 both13 Elascsearch13 and13 third-shy‐party13 tools13 at13 the13 same13 instant13 using13 the13 output13 secon13 of13 the13 Logstash13 configuraon13 Any13 alerts13 are13 generated13 during13 the13 filter13 secon13 processing13 of13 the13 Logstash13 configuraon13

Create13 alerts13 in13 Logstash13 The13 Logstash13 filter13 secon13 is13 used13 to13 trigger13 alerts13 based13 on13 the13 log13 records13 being13 processed13 For13 example13 log13 records13 can13 be13 checked13 for13 errors13 warnings13 crashed13 or13 stopped13 messages13 and13 generate13 an13 alert13

Here13 is13 an13 example13 Logstash13 filter13 code13

13 if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

The13 specific13 tags13 are13 set13 against13 the13 log13 records13 to13 trigger13 the13 alert13 later13 in13 the13 configuraon13 in13 the13 output13 secon13

Control13 the13 number13 of13 generated13 alerts13 using13 the13 Logstash13 throcle13 plugin13 so13 that13 either13 a13 lot13 of13 instances13 in13 a13 fixed13 me13 period13 are13 required13 before13 an13 alert13 is13 sent13 or13 a13 specific13 number13 of13 instances13 triggers13 the13 alert13

Page of 36 71 Page of 36 71

Edit13 the13 plugin13 parameters13 to13 meet13 system13 requirements13

13 13 13 13 13 if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 10

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle13

after_count =gt 20

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 30

period =gt 1200

key =gt servicename_alarm_warning

Page of 37 71 Page of 37 71

add_tag =gt servicename_throttled

13 13 13 13 13 13 13 13 13

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 40

period =gt 1500

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

Enrich13 alerts13 with13 message13 detail13 Some13 of13 the13 third-shy‐party13 integraons13 benefit13 from13 having13 components13 of13 the13 raw13 log13 record13 sent13 in13 addion13 to13 the13 raw13 record13 or13 fields13 in13 the13 record13 edited13 transformed13 or13 translated13

Grok13 Logstash13 allows13 the13 log13 record13 to13 be13 broken13 up13 into13 logical13 secons13 and13 each13 secon13 stored13 in13 a13 new13 parameter13 using13 the13 grok13 parameter13 in13 the13 filter13 secon13 Here13 is13 an13 example13 grok13

13 grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

To13 build13 the13 grok13 statement13 Logstash13 offers13 an13 online13 tool13 at13 hcpsgrokdebugherokuappcom13

Translate13 If13 specific13 fields13 in13 the13 log13 record13 are13 coded13 (for13 example13 the13 Bluemix13 app13 name13 is13 oaen13 not13 sent13 in13 the13 syslog13 messages13 instead13 the13 GUID13 is13 sent13 which13 is13 not13 useful)13 then13 the13 filter13 secon13 translate13 plugin13 can13 be13 used13 to13 add13 new13 fields13 by13 holding13 the13 decoded13 versions13 of13 those13 values13

13 translate

field =gt data1

destination =gt data1

override =gt true

Page of 38 71 Page of 38 71

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

dc50f6e4-c052-436a-bf27-95ba392c4228 monverifyservicemanagement

0c677e82-5bfc-4e38-9155-e416249e3ee6 ms-catalog-postpyretic-pomiculture

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

However13 the13 translate13 plugin13 is13 not13 included13 with13 the13 Logstash13 installaon13 In13 the13 next13 secon13 I13 will13 13 describe13 how13 to13 add13 new13 plugins13 to13 the13 Logstash13 instance13

Add13 custom13 plugins13 to13 Logstash13 As13 an13 open13 source13 tool13 Logstash13 encourages13 users13 to13 develop13 content13 for13 all13 users13 to13 share13 Some13 of13 the13 plugins13 that13 were13 created13 by13 our13 community13 include13 a13 translaon13 plugin13 an13 output13 Slack13 plugin13 and13 an13 output13 syslog13 plugin13 Letrsquos13 look13 at13 how13 to13 add13 these13 plugins13 to13 your13 environment13

Filter13 translate13 plugin13 To13 add13 the13 translate13 plugin13

1 Download13 the13 plugin13

2 Copy13 the13 plugin13 file13 logstash-shy‐filter-shy‐translate-shy‐214gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-filter-translate 214

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Output13 Slack13 plugin13 To13 add13 the13 Slack13 plugin13

Page of 39 71 Page of 39 71

1 Download13 the13 plugin13 13

2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐slack-shy‐014gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-output-slack 014

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Output13 syslog13 plugin13 To13 add13 the13 syslog13 plugin13

1 Download13 the13 plugin13 13

2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐syslog-shy‐214gem13 to13 the13 ELK13 Virtual13 Server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-output-syslog 214

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Integrate13 Logstash13 with13 Slack13 To13 send13 generated13 alerts13 to13 Slack13 complete13 these13 steps13

1 Create13 a13 Slack13 integraon13 13

2 Use13 the13 Slack13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 Slack13 integraon13

113 Login13 to13 Slack13 and13 go13 to13 the13 Apps13 page13 13

Page of 40 71 Page of 40 71

13

313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13

13

A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13

13

Page of 42 71 Page of 42 71

The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13

13

The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13

13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

slack

url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5

channel =gt cloud-operations-mon

username =gt nick_cawood

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13

Page of 43 71 Page of 43 71

Page of 44 71 Page of 44 71

In13 Slack13 the13 alerts13 look13 like13 this13

13

Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13

1 Create13 a13 PagerDuty13 integraon13 13

2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13

1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13

13

2 Click13 Add13 New13 Service13

13

313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13

Page of 45 71 Page of 45 71

13

413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13

513 Click13 Add13 Service13

13

The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13

Page of 46 71 Page of 46 71

13

The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt be7eb1ea9c224018923e818d3e891d0b

incident_key =gt logstashhosttype

Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13

Page of 47 71 Page of 47 71

13

Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13

1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13

2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13

3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13

4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13

13 13 http

url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo

http_method =gt post

format =gt json

For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13

13 13 syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

Page of 48 71 Page of 48 71

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13

13

The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13

if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2

This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13

Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13

Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13

Logstash13 severity Shown13 in13 Omnibus13 as

alert Indeterminate

crical Warning

debug 7

emergency Clear

error Minor

informaonal 6

noce Crical

warning Major

Page of 49 71 Page of 49 71

13

Page of 50 71 Page of 50 71

Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13

input

tcp

port =gt 5000

type =gt syslog

filter

mutate

add_field =gt format =gt cf

grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

translate

field =gt data1

destination =gt data1

override =gt true

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

=== Check Alarm Criteria ===

Standard Key Words

if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

=== Alarm Exclusions ===

if [message] =~ (i)sometexthere

mutate remove_tag =gt servicename_alarm

Throttle alarms so we do not hit PD rate limits

Page of 52 71 Page of 52 71

if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 5

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle

after_count =gt 5

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 5

period =gt 1200

key =gt servicename_alarm_warning

add_tag =gt servicename_throttled

Page of 53 71 Page of 53 71

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

output

elasticsearch

hosts =gt localhost

stdout codec =gt rubydebug

if servicename_alarm_err in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

Page of 54 71 Page of 54 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 55 71 Page of 55 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

Page of 56 71 Page of 56 71

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert err message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_error in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

Page of 57 71 Page of 57 71

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 58 71 Page of 58 71

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 59 71 Page of 59 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert error message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

Page of 60 71 Page of 60 71

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

Page of 61 71 Page of 61 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 62 71 Page of 62 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt x

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert fatal message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

Page of 63 71 Page of 63 71

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

Page of 64 71 Page of 64 71

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 65 71 Page of 65 71

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt heavy_exclamation_mark

Page of 66 71 Page of 66 71

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert WARNING message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 67 71 Page of 67 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

Page of 68 71 Page of 68 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 69 71 Page of 69 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

Page of 70 71 Page of 70 71

Page of 71 71 Page of 71 71

Page 16: How to integrate the Elasticsearch Logstash Kibana (ELK ... · How to integrate the Elasticsearch Logstash Kibana (ELK) log analytics stack into IBM Bluemix Nick Cawood IBM Cloud

4 Edit13 the13 configkibanayml13 file13 so13 the13 contents13 are13

Kibana is served by a back end server This controls which port to use

port 5601

The host to bind the server to

host 0000

If you are running kibana behind a proxy and want to mount it at a path

specify that path here The basePath cant end in a slash

serverbasePath

The maximum payload size in bytes on incoming server requests

servermaxPayloadBytes 1048576

The Elasticsearch instance to use for all your queries

elasticsearchurl httplocalhost9200

preserve_elasticsearch_host true will send the hostname specified in `elasticsearch` If you set it to false

then the host you use to connect to this Kibana instance will be sent

elasticsearchpreserveHost true

Kibana uses an index in Elasticsearch to store saved searches visualizations

and dashboards It will create a new index if it doesnt already exist

kibanaindex kibana

The default application to load

kibanadefaultAppId discover

If your Elasticsearch is protected with basic auth these are the user credentials

used by the Kibana server to perform maintenance on the kibana_index at startup Your

Kibana users will still need to authenticate with Elasticsearch (which is proxied through

the Kibana server)

Page of 16 71 Page of 16 71

elasticsearchusername user

elasticsearchpassword pass

SSL for outgoing requests from the Kibana Server to the browser (PEM formatted)

serversslcert pathtoyourservercrt

serversslkey pathtoyourserverkey

Optional setting to validate that your Elasticsearch backend uses the same key files (PEM formatted)

elasticsearchsslcert pathtoyourclientcrt

elasticsearchsslkey pathtoyourclientkey

If you need to provide a CA certificate for your Elasticsearch instance put

the path of the pem file here

elasticsearchsslca pathtoyourCApem

Set to false to have a complete disregard for the validity of the SSL

certificate

elasticsearchsslverify true

Time in milliseconds to wait for elasticsearch to respond to pings defaults to

request_timeout setting

elasticsearchpingTimeout 1500

Time in milliseconds to wait for responses from the back end or elasticsearch

This must be gt 0

elasticsearchrequestTimeout 30000

Time in milliseconds for Elasticsearch to wait for responses from shards

Set to 0 to disable

elasticsearchshardTimeout 0

Time in milliseconds to wait for Elasticsearch at Kibana startup before retrying

elasticsearchstartupTimeout 5000

elasticsearchstartupTimeout 30000

Page of 17 71 Page of 17 71

Set the path to where you would like the process id file to be created

pidfile varrunkibanapid

If you would like to send the log output to a file you can set the path below

loggingdest stdout

Set this to true to suppress all logging output

loggingsilent false

Set this to true to suppress all logging output except for error messages

loggingquiet false

Set this to true to log all events including system usage information and all requests

loggingverbose false

Kibana13 is13 now13 installed13

Start13 Kibana13 To13 start13 Kibana13 go13 to13 the13 bin13 directory13 and13 run13 the13 kibana13 executable13

[rootelkforcase]$

[rootelkforcase]$ cd homekibanakibana-451-linux-x64bin

[rootelkforcase]$ kibana amp

[1] 22283

[rootelkforcase] log [095900542] [info][status][pluginkibana] Status changed from uninitialized to green - Ready

log [095900600] [info][status][pluginelasticsearch] Status changed from uninitialized to yellow - Waiting for Elasticsearch

log [095900689] [info][status][pluginkbn_vislib_vis_types] Status changed from uninitialized to green - Ready

log [095900717] [info][status][pluginmarkdown_vis] Status changed from uninitialized to green - Ready

log [095900721] [info][status][pluginmetric_vis] Status changed from uninitialized to green - Ready

log [095900814] [info][status][pluginspyModes] Status changed from uninitialized to green - Ready

log [095900934] [info][status][pluginstatusPage] Status changed from uninitialized to green - Ready

Page of 18 71 Page of 18 71

log [095900941] [info][status][plugintable_vis] Status changed from uninitialized to green - Ready

log [095901067] [info][listening] Server running at http00005601

log [095906083] [info][status][pluginelasticsearch] Status changed from yellow to yellow - No existing Kibana index found

log [095908942] [info][status][pluginelasticsearch] Status changed from yellow to green - Kibana index ready

Kibana13 is13 now13 started13

Broadcast13 Bluemix13 app13 log13 records13 Bluemix13 apps13 (which13 are13 shown13 in13 the13 US13 South13 Bluemix13 region13 of13 the13 System13 Architecture13 diagram)13 will13 produce13 logs13 in13 a13 syslog13 rfc542413 format13 using13 a13 Bluemix13 tool13 called13 Loggregator13

For13 Logstash13 to13 receive13 these13 logs13 each13 Bluemix13 app13 needs13 to13 broadcast13 these13 logs13 To13 do13 that13 run13 the13 following13 commands13 on13 each13 Bluemix13 app13 using13 the13 public13 IP13 address13 of13 the13 ELK13 virtual13 server13 the13 ELK13 tag13 to13 trace13 the13 process13 and13 insert13 the13 name13 of13 each13 app13 in13 turn13

cf cups elk -l syslogltpublicIPaddress5000 cf bind-service ltmyappgt elk

cf restage ltmyappgt

Install13 Logstash13 To13 install13 Logstash13 as13 root13 user13 create13 a13 directory13 for13 the13 installaon13 and13 untar13 the13 installaon13 file13 there13

13 13 [rootelkforcase]$ cd home

[rootelkforcase]$ mkdir logstash

[rootelkforcase]$ cp homeibmcloudlogstash-233targz homelogstash

[rootelkforcase]$ cd logstash

[rootelkforcase]$ gunzip logstash-233targz

[rootelkforcase]$ untar logstash-233tar

Write13 Logstash13 configura0on13 Logstash13 requires13 a13 configuraon13 file13 to13 be13 wricen13 to13 receive13 transform13 and13 output13 the13 log13 records13 to13 Elascsearch13 This13 secon13 explains13 the13 basic13 requirements13 of13 processing13 Bluemix13 app13 log13 records13 into13 Elascsearch13 (more13 complex13 configuraons13 are13 detailed13 later13 in13 this13 document13 for13 the13 integraons13 with13 third-shy‐party13 tools)13

Create13 a13 conf13 file13 as13 root13 user13

[rootelkforcase]$ cd homelogstashlogstash-233

[rootelkforcase]$ mkdir config

Page of 19 71 Page of 19 71

[rootelkforcase]$ cd config

[rootelkforcase]$ touch loggregatorconf

Edit13 the13 loggregator13 file13 with13 the13 following13 contents13

13 Input13 secon13 defines13 which13 log13 records13 to13 receive13

13 port13 and13 type13 reflect13 method13 selected13 for13 Cloud13 Foundry13 to13 broadcast13 the13 records13

input

tcp

port =gt 5000

type =gt syslog

13 Filter13 secon13 ndash13 transforms13 the13 data13

filter

Add a field to identify the records as Cloud Foundry based

mutate

add_field =gt format =gt cf

13 Output13 secon13 ndash13 sends13 the13 data13 to13 its13 desnaon(s)13

output

13 Use13 the13 elascsearch13 plugin13 to13 send13 the13 data13 to13 the13 elascsearch13 instance13

13 elasticsearch

hosts =gt localhost

stdout codec =gt rubydebug

13 End13 of13 Config13

Save13 the13 file13 Verify13 the13 format13 of13 the13 file13 is13 acceptable13 by13 running13 the13 following13 commands13

[rootelkforcase]$ cd homelogstashlogstash-233

[rootelkforcase]$ binlogstash agent --configtest -f configloggregatorconf

Page of 20 71 Page of 20 71

Configuraon13 is13 OK13

Start13 Logstash13 Aaer13 you13 have13 wricen13 and13 verified13 the13 Logstash13 configuraon13 you13 can13 start13 Logstash13 to13 process13 records13 from13 the13 Bluemix13 apps13 (via13 Cloud13 Foundry)13 Run13 the13 following13 as13 root13 user13

[rootelkforcase]$ cd homelogstashlogstash-233

[rootelkforcase]$ binlogstash agent -f configloggregatorconf

Logstash13 is13 now13 started13

Monitor13 Logstash13 processing13 Once13 Logstash13 is13 running13 and13 log13 records13 are13 being13 produced13 by13 the13 Bluemix13 apps13 you13 can13 monitor13 progress13 in13 the13 same13 window13 that13 was13 used13 to13 start13 Logstash13 Output13 is13 similar13 to13 the13 following13

13

The13 log13 records13 being13 broadcast13 from13 the13 Bluemix13 apps13 are13 now13 sent13 to13 Elascsearch13 from13 the13 Logstash13 output13 configuraon13 Records13 are13 sent13 in13 real13 me13 If13 Logstash13 is13 not13 running13 while13 log13 records13 are13 being13 produced13 then13 these13 records13 will13 not13 be13 processed13 by13 Logstash13 and13 will13 not13 be13 stored13 in13 Elascsearch13

Page of 21 71 Page of 21 71

Use13 ELK13 The13 log13 records13 from13 the13 Bluemix13 apps13 are13 now13 being13 stored13 and13 indexed13 by13 the13 Elascsearch13 component13 This13 data13 can13 be13 accessed13 via13 the13 Kibana13 applicaon13

Kibana13 Discover13 page13 Using13 a13 web13 browser13 either13 directly13 from13 a13 client13 or13 using13 the13 VNC13 client13 access13 Kibana13 using13 the13 publicIPaddress13 of13 the13 ELK13 virtual13 server13

hcpltpublicIPaddressgt560113

No13 userpassword13 credenals13 are13 required13

Kibana13 defaults13 to13 a13 search13 of13 all13 records13 ()13 in13 the13 last13 1513 minutes13 and13 returns13 the13 results13

13

More13 complex13 searches13 are13 possible13 by13 changing13 the13 me13 filter13 and13 supplying13 search13 queries13

Change13 the13 0me13 filter13 To13 change13 the13 me13 ranges13 click13 the13 clock13 symbol13 in13 the13 top-shy‐right13 of13 the13 Kibana13 Discover13 page13

13

You13 will13 see13 the13 following13 opons13

13

Page of 22 71 Page of 22 71

A13 variety13 of13 quick13 me13 ranges13 can13 be13 selected13 or13 you13 can13 switch13 to13 relave13 or13 absolute13 me13 ranges13

Quick13 me13 ranges13 allow13 switching13 between13 last13 X13 minutes13 to13 last13 Y13 hours13 or13 Z13 days13 etc13

Relave13 me13 ranges13 allow13 more13 specific13 versions13 of13 the13 Quick13 opons13 from13 specific13 minuteshoursdays13 to13 now13

13

Absolute13 me13 ranges13 specify13 the13 start13 and13 end13 mes13 for13 the13 searches13

13

Custom13 search13 queries13 You13 can13 use13 the13 text13 window13 of13 the13 Discover13 page13 to13 search13 for13 matches13 within13 the13 log13 records13 You13 can13 use13 Apache13 Lucene13 or13 Elascsearch13 query13 DSL13 syntax13

13

Page of 23 71 Page of 23 71

Saved13 searches13 You13 can13 save13 any13 custom13 search13 syntax13 and13 choice13 of13 me13 range13 for13 future13 ease13 of13 use13 Any13 relave13 or13 absolute13 me13 frame13 is13 saved13 as13 part13 of13 the13 search13 13

To13 save13 a13 search13

1 Set13 up13 and13 run13 the13 required13 syntax13 and13 me13 frame13

13

2 Click13 the13 Save13 Search13 icon13

13

3 Type13 in13 a13 name13 for13 the13 saved13 search13

Page of 24 71 Page of 24 71

13

7 To13 delete13 or13 update13 saved13 searches13 click13 13 manage13 saved13 searches13

13

Use13 the13 edit13 icons13 to13 make13 any13 changes13

Page of 26 71 Page of 26 71

13

Build13 simple13 Kibana13 histograms13 Kibana13 allows13 you13 to13 build13 charts13 and13 dashboards13 based13 on13 the13 search13 capability13 offered13 by13 the13 Discover13 page13 Saved13 searches13 can13 be13 used13 as13 the13 basis13 for13 charts13 and13 dashboards13 built13 using13 those13 charts13 To13 build13 a13 simple13 histogram13

1 Select13 the13 Visualize13 page13

13

2 Select13 Ver0cal13 Bar13 Chart13

13

3 To13 use13 a13 saved13 search13 click13 From13 a13 saved13 search13

Page of 27 71 Page of 27 71

13

4 Select13 a13 previously13 saved13 search13

13

5 Select13 to13 edit13 the13 x-shy‐axis

6 Edit13 the13 fields13 with13 informaon13 about13 the13 aggregaon13 field13 interval13 and13 custom13 label13

Page of 28 71 Page of 28 71

13

Page of 29 71 Page of 29 71

7 Click13 the13 green13 arrow13 The13 histogram13 is13 created13 and13 run13 based13 on13 the13 saved13 search

13

8 To13 save13 the13 histogram13 click13 the13 Save13 Visualiza0on13 icon13 and13 add13 a13 tle13

13

13

9 Click13 Save13

10 To13 view13 the13 visualisaon13 click13 the13 Dashboard13 page13

Page of 30 71 Page of 30 71

13

11 Click13 the13 Add13 Visualiza0on13 icon13

13

12 Select13 the13 visiualizaon13 that13 you13 just13 created13

13

Page of 31 71 Page of 31 71

13

13 To13 save13 the13 dashboard13 click13 the13 Save13 Dashboard13 icon13

Page of 32 71 Page of 32 71

13

14 Add13 a13 tle13 and13 select13 whether13 to13 store13 the13 selected13 me13 frame13 with13 the13 dashboard13

13

15 Click13 Save13

16 You13 can13 edit13 the13 shape13 and13 number13 of13 charts13 and13 save13 the13 dashboard13

17 To13 re-shy‐run13 the13 dashboard13 select13 the13 Dashboard13 page13 and13 click13 the13 Load13 Saved13 Dashboard13 icon13

Page of 33 71 Page of 33 71

13

18 Select13 the13 saved13 dashboard13 from13 the13 list13

13

19 The13 dashboard13 runs13

13

Page of 34 71 Page of 34 71

Page of 35 71 Page of 35 71

Aler0ng13 with13 ELK13 ELK13 examines13 the13 log13 records13 in13 process13 to13 send13 alerts13 to13 third-shy‐party13 alerng13 event13 management13 and13 social13 media13 applicaons13 As13 shown13 previously13 in13 the13 ELK13 architecture13 the13 ELK13 strategy13 for13 alerng13 is13 to13 send13 alerts13 from13 the13 Logstash13 component13 directly13 Therefore13 log13 records13 and13 alerts13 are13 sent13 to13 both13 Elascsearch13 and13 third-shy‐party13 tools13 at13 the13 same13 instant13 using13 the13 output13 secon13 of13 the13 Logstash13 configuraon13 Any13 alerts13 are13 generated13 during13 the13 filter13 secon13 processing13 of13 the13 Logstash13 configuraon13

Create13 alerts13 in13 Logstash13 The13 Logstash13 filter13 secon13 is13 used13 to13 trigger13 alerts13 based13 on13 the13 log13 records13 being13 processed13 For13 example13 log13 records13 can13 be13 checked13 for13 errors13 warnings13 crashed13 or13 stopped13 messages13 and13 generate13 an13 alert13

Here13 is13 an13 example13 Logstash13 filter13 code13

13 if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

The13 specific13 tags13 are13 set13 against13 the13 log13 records13 to13 trigger13 the13 alert13 later13 in13 the13 configuraon13 in13 the13 output13 secon13

Control13 the13 number13 of13 generated13 alerts13 using13 the13 Logstash13 throcle13 plugin13 so13 that13 either13 a13 lot13 of13 instances13 in13 a13 fixed13 me13 period13 are13 required13 before13 an13 alert13 is13 sent13 or13 a13 specific13 number13 of13 instances13 triggers13 the13 alert13

Page of 36 71 Page of 36 71

Edit13 the13 plugin13 parameters13 to13 meet13 system13 requirements13

13 13 13 13 13 if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 10

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle13

after_count =gt 20

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 30

period =gt 1200

key =gt servicename_alarm_warning

Page of 37 71 Page of 37 71

add_tag =gt servicename_throttled

13 13 13 13 13 13 13 13 13

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 40

period =gt 1500

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

Enrich13 alerts13 with13 message13 detail13 Some13 of13 the13 third-shy‐party13 integraons13 benefit13 from13 having13 components13 of13 the13 raw13 log13 record13 sent13 in13 addion13 to13 the13 raw13 record13 or13 fields13 in13 the13 record13 edited13 transformed13 or13 translated13

Grok13 Logstash13 allows13 the13 log13 record13 to13 be13 broken13 up13 into13 logical13 secons13 and13 each13 secon13 stored13 in13 a13 new13 parameter13 using13 the13 grok13 parameter13 in13 the13 filter13 secon13 Here13 is13 an13 example13 grok13

13 grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

To13 build13 the13 grok13 statement13 Logstash13 offers13 an13 online13 tool13 at13 hcpsgrokdebugherokuappcom13

Translate13 If13 specific13 fields13 in13 the13 log13 record13 are13 coded13 (for13 example13 the13 Bluemix13 app13 name13 is13 oaen13 not13 sent13 in13 the13 syslog13 messages13 instead13 the13 GUID13 is13 sent13 which13 is13 not13 useful)13 then13 the13 filter13 secon13 translate13 plugin13 can13 be13 used13 to13 add13 new13 fields13 by13 holding13 the13 decoded13 versions13 of13 those13 values13

13 translate

field =gt data1

destination =gt data1

override =gt true

Page of 38 71 Page of 38 71

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

dc50f6e4-c052-436a-bf27-95ba392c4228 monverifyservicemanagement

0c677e82-5bfc-4e38-9155-e416249e3ee6 ms-catalog-postpyretic-pomiculture

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

However13 the13 translate13 plugin13 is13 not13 included13 with13 the13 Logstash13 installaon13 In13 the13 next13 secon13 I13 will13 13 describe13 how13 to13 add13 new13 plugins13 to13 the13 Logstash13 instance13

Add13 custom13 plugins13 to13 Logstash13 As13 an13 open13 source13 tool13 Logstash13 encourages13 users13 to13 develop13 content13 for13 all13 users13 to13 share13 Some13 of13 the13 plugins13 that13 were13 created13 by13 our13 community13 include13 a13 translaon13 plugin13 an13 output13 Slack13 plugin13 and13 an13 output13 syslog13 plugin13 Letrsquos13 look13 at13 how13 to13 add13 these13 plugins13 to13 your13 environment13

Filter13 translate13 plugin13 To13 add13 the13 translate13 plugin13

1 Download13 the13 plugin13

2 Copy13 the13 plugin13 file13 logstash-shy‐filter-shy‐translate-shy‐214gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-filter-translate 214

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Output13 Slack13 plugin13 To13 add13 the13 Slack13 plugin13

Page of 39 71 Page of 39 71

1 Download13 the13 plugin13 13

2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐slack-shy‐014gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-output-slack 014

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Output13 syslog13 plugin13 To13 add13 the13 syslog13 plugin13

1 Download13 the13 plugin13 13

2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐syslog-shy‐214gem13 to13 the13 ELK13 Virtual13 Server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-output-syslog 214

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Integrate13 Logstash13 with13 Slack13 To13 send13 generated13 alerts13 to13 Slack13 complete13 these13 steps13

1 Create13 a13 Slack13 integraon13 13

2 Use13 the13 Slack13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 Slack13 integraon13

113 Login13 to13 Slack13 and13 go13 to13 the13 Apps13 page13 13

Page of 40 71 Page of 40 71

13

313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13

13

A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13

13

Page of 42 71 Page of 42 71

The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13

13

The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13

13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

slack

url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5

channel =gt cloud-operations-mon

username =gt nick_cawood

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13

Page of 43 71 Page of 43 71

Page of 44 71 Page of 44 71

In13 Slack13 the13 alerts13 look13 like13 this13

13

Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13

1 Create13 a13 PagerDuty13 integraon13 13

2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13

1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13

13

2 Click13 Add13 New13 Service13

13

313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13

Page of 45 71 Page of 45 71

13

413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13

513 Click13 Add13 Service13

13

The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13

Page of 46 71 Page of 46 71

13

The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt be7eb1ea9c224018923e818d3e891d0b

incident_key =gt logstashhosttype

Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13

Page of 47 71 Page of 47 71

13

Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13

1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13

2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13

3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13

4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13

13 13 http

url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo

http_method =gt post

format =gt json

For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13

13 13 syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

Page of 48 71 Page of 48 71

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13

13

The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13

if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2

This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13

Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13

Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13

Logstash13 severity Shown13 in13 Omnibus13 as

alert Indeterminate

crical Warning

debug 7

emergency Clear

error Minor

informaonal 6

noce Crical

warning Major

Page of 49 71 Page of 49 71

13

Page of 50 71 Page of 50 71

Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13

input

tcp

port =gt 5000

type =gt syslog

filter

mutate

add_field =gt format =gt cf

grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

translate

field =gt data1

destination =gt data1

override =gt true

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

=== Check Alarm Criteria ===

Standard Key Words

if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

=== Alarm Exclusions ===

if [message] =~ (i)sometexthere

mutate remove_tag =gt servicename_alarm

Throttle alarms so we do not hit PD rate limits

Page of 52 71 Page of 52 71

if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 5

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle

after_count =gt 5

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 5

period =gt 1200

key =gt servicename_alarm_warning

add_tag =gt servicename_throttled

Page of 53 71 Page of 53 71

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

output

elasticsearch

hosts =gt localhost

stdout codec =gt rubydebug

if servicename_alarm_err in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

Page of 54 71 Page of 54 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 55 71 Page of 55 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

Page of 56 71 Page of 56 71

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert err message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_error in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

Page of 57 71 Page of 57 71

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 58 71 Page of 58 71

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 59 71 Page of 59 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert error message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

Page of 60 71 Page of 60 71

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

Page of 61 71 Page of 61 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 62 71 Page of 62 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt x

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert fatal message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

Page of 63 71 Page of 63 71

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

Page of 64 71 Page of 64 71

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 65 71 Page of 65 71

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt heavy_exclamation_mark

Page of 66 71 Page of 66 71

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert WARNING message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 67 71 Page of 67 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

Page of 68 71 Page of 68 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 69 71 Page of 69 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

Page of 70 71 Page of 70 71

Page of 71 71 Page of 71 71

Page 17: How to integrate the Elasticsearch Logstash Kibana (ELK ... · How to integrate the Elasticsearch Logstash Kibana (ELK) log analytics stack into IBM Bluemix Nick Cawood IBM Cloud

elasticsearchusername user

elasticsearchpassword pass

SSL for outgoing requests from the Kibana Server to the browser (PEM formatted)

serversslcert pathtoyourservercrt

serversslkey pathtoyourserverkey

Optional setting to validate that your Elasticsearch backend uses the same key files (PEM formatted)

elasticsearchsslcert pathtoyourclientcrt

elasticsearchsslkey pathtoyourclientkey

If you need to provide a CA certificate for your Elasticsearch instance put

the path of the pem file here

elasticsearchsslca pathtoyourCApem

Set to false to have a complete disregard for the validity of the SSL

certificate

elasticsearchsslverify true

Time in milliseconds to wait for elasticsearch to respond to pings defaults to

request_timeout setting

elasticsearchpingTimeout 1500

Time in milliseconds to wait for responses from the back end or elasticsearch

This must be gt 0

elasticsearchrequestTimeout 30000

Time in milliseconds for Elasticsearch to wait for responses from shards

Set to 0 to disable

elasticsearchshardTimeout 0

Time in milliseconds to wait for Elasticsearch at Kibana startup before retrying

elasticsearchstartupTimeout 5000

elasticsearchstartupTimeout 30000

Page of 17 71 Page of 17 71

Set the path to where you would like the process id file to be created

pidfile varrunkibanapid

If you would like to send the log output to a file you can set the path below

loggingdest stdout

Set this to true to suppress all logging output

loggingsilent false

Set this to true to suppress all logging output except for error messages

loggingquiet false

Set this to true to log all events including system usage information and all requests

loggingverbose false

Kibana13 is13 now13 installed13

Start13 Kibana13 To13 start13 Kibana13 go13 to13 the13 bin13 directory13 and13 run13 the13 kibana13 executable13

[rootelkforcase]$

[rootelkforcase]$ cd homekibanakibana-451-linux-x64bin

[rootelkforcase]$ kibana amp

[1] 22283

[rootelkforcase] log [095900542] [info][status][pluginkibana] Status changed from uninitialized to green - Ready

log [095900600] [info][status][pluginelasticsearch] Status changed from uninitialized to yellow - Waiting for Elasticsearch

log [095900689] [info][status][pluginkbn_vislib_vis_types] Status changed from uninitialized to green - Ready

log [095900717] [info][status][pluginmarkdown_vis] Status changed from uninitialized to green - Ready

log [095900721] [info][status][pluginmetric_vis] Status changed from uninitialized to green - Ready

log [095900814] [info][status][pluginspyModes] Status changed from uninitialized to green - Ready

log [095900934] [info][status][pluginstatusPage] Status changed from uninitialized to green - Ready

Page of 18 71 Page of 18 71

log [095900941] [info][status][plugintable_vis] Status changed from uninitialized to green - Ready

log [095901067] [info][listening] Server running at http00005601

log [095906083] [info][status][pluginelasticsearch] Status changed from yellow to yellow - No existing Kibana index found

log [095908942] [info][status][pluginelasticsearch] Status changed from yellow to green - Kibana index ready

Kibana13 is13 now13 started13

Broadcast13 Bluemix13 app13 log13 records13 Bluemix13 apps13 (which13 are13 shown13 in13 the13 US13 South13 Bluemix13 region13 of13 the13 System13 Architecture13 diagram)13 will13 produce13 logs13 in13 a13 syslog13 rfc542413 format13 using13 a13 Bluemix13 tool13 called13 Loggregator13

For13 Logstash13 to13 receive13 these13 logs13 each13 Bluemix13 app13 needs13 to13 broadcast13 these13 logs13 To13 do13 that13 run13 the13 following13 commands13 on13 each13 Bluemix13 app13 using13 the13 public13 IP13 address13 of13 the13 ELK13 virtual13 server13 the13 ELK13 tag13 to13 trace13 the13 process13 and13 insert13 the13 name13 of13 each13 app13 in13 turn13

cf cups elk -l syslogltpublicIPaddress5000 cf bind-service ltmyappgt elk

cf restage ltmyappgt

Install13 Logstash13 To13 install13 Logstash13 as13 root13 user13 create13 a13 directory13 for13 the13 installaon13 and13 untar13 the13 installaon13 file13 there13

13 13 [rootelkforcase]$ cd home

[rootelkforcase]$ mkdir logstash

[rootelkforcase]$ cp homeibmcloudlogstash-233targz homelogstash

[rootelkforcase]$ cd logstash

[rootelkforcase]$ gunzip logstash-233targz

[rootelkforcase]$ untar logstash-233tar

Write13 Logstash13 configura0on13 Logstash13 requires13 a13 configuraon13 file13 to13 be13 wricen13 to13 receive13 transform13 and13 output13 the13 log13 records13 to13 Elascsearch13 This13 secon13 explains13 the13 basic13 requirements13 of13 processing13 Bluemix13 app13 log13 records13 into13 Elascsearch13 (more13 complex13 configuraons13 are13 detailed13 later13 in13 this13 document13 for13 the13 integraons13 with13 third-shy‐party13 tools)13

Create13 a13 conf13 file13 as13 root13 user13

[rootelkforcase]$ cd homelogstashlogstash-233

[rootelkforcase]$ mkdir config

Page of 19 71 Page of 19 71

[rootelkforcase]$ cd config

[rootelkforcase]$ touch loggregatorconf

Edit13 the13 loggregator13 file13 with13 the13 following13 contents13

13 Input13 secon13 defines13 which13 log13 records13 to13 receive13

13 port13 and13 type13 reflect13 method13 selected13 for13 Cloud13 Foundry13 to13 broadcast13 the13 records13

input

tcp

port =gt 5000

type =gt syslog

13 Filter13 secon13 ndash13 transforms13 the13 data13

filter

Add a field to identify the records as Cloud Foundry based

mutate

add_field =gt format =gt cf

13 Output13 secon13 ndash13 sends13 the13 data13 to13 its13 desnaon(s)13

output

13 Use13 the13 elascsearch13 plugin13 to13 send13 the13 data13 to13 the13 elascsearch13 instance13

13 elasticsearch

hosts =gt localhost

stdout codec =gt rubydebug

13 End13 of13 Config13

Save13 the13 file13 Verify13 the13 format13 of13 the13 file13 is13 acceptable13 by13 running13 the13 following13 commands13

[rootelkforcase]$ cd homelogstashlogstash-233

[rootelkforcase]$ binlogstash agent --configtest -f configloggregatorconf

Page of 20 71 Page of 20 71

Configuraon13 is13 OK13

Start13 Logstash13 Aaer13 you13 have13 wricen13 and13 verified13 the13 Logstash13 configuraon13 you13 can13 start13 Logstash13 to13 process13 records13 from13 the13 Bluemix13 apps13 (via13 Cloud13 Foundry)13 Run13 the13 following13 as13 root13 user13

[rootelkforcase]$ cd homelogstashlogstash-233

[rootelkforcase]$ binlogstash agent -f configloggregatorconf

Logstash13 is13 now13 started13

Monitor13 Logstash13 processing13 Once13 Logstash13 is13 running13 and13 log13 records13 are13 being13 produced13 by13 the13 Bluemix13 apps13 you13 can13 monitor13 progress13 in13 the13 same13 window13 that13 was13 used13 to13 start13 Logstash13 Output13 is13 similar13 to13 the13 following13

13

The13 log13 records13 being13 broadcast13 from13 the13 Bluemix13 apps13 are13 now13 sent13 to13 Elascsearch13 from13 the13 Logstash13 output13 configuraon13 Records13 are13 sent13 in13 real13 me13 If13 Logstash13 is13 not13 running13 while13 log13 records13 are13 being13 produced13 then13 these13 records13 will13 not13 be13 processed13 by13 Logstash13 and13 will13 not13 be13 stored13 in13 Elascsearch13

Page of 21 71 Page of 21 71

Use13 ELK13 The13 log13 records13 from13 the13 Bluemix13 apps13 are13 now13 being13 stored13 and13 indexed13 by13 the13 Elascsearch13 component13 This13 data13 can13 be13 accessed13 via13 the13 Kibana13 applicaon13

Kibana13 Discover13 page13 Using13 a13 web13 browser13 either13 directly13 from13 a13 client13 or13 using13 the13 VNC13 client13 access13 Kibana13 using13 the13 publicIPaddress13 of13 the13 ELK13 virtual13 server13

hcpltpublicIPaddressgt560113

No13 userpassword13 credenals13 are13 required13

Kibana13 defaults13 to13 a13 search13 of13 all13 records13 ()13 in13 the13 last13 1513 minutes13 and13 returns13 the13 results13

13

More13 complex13 searches13 are13 possible13 by13 changing13 the13 me13 filter13 and13 supplying13 search13 queries13

Change13 the13 0me13 filter13 To13 change13 the13 me13 ranges13 click13 the13 clock13 symbol13 in13 the13 top-shy‐right13 of13 the13 Kibana13 Discover13 page13

13

You13 will13 see13 the13 following13 opons13

13

Page of 22 71 Page of 22 71

A13 variety13 of13 quick13 me13 ranges13 can13 be13 selected13 or13 you13 can13 switch13 to13 relave13 or13 absolute13 me13 ranges13

Quick13 me13 ranges13 allow13 switching13 between13 last13 X13 minutes13 to13 last13 Y13 hours13 or13 Z13 days13 etc13

Relave13 me13 ranges13 allow13 more13 specific13 versions13 of13 the13 Quick13 opons13 from13 specific13 minuteshoursdays13 to13 now13

13

Absolute13 me13 ranges13 specify13 the13 start13 and13 end13 mes13 for13 the13 searches13

13

Custom13 search13 queries13 You13 can13 use13 the13 text13 window13 of13 the13 Discover13 page13 to13 search13 for13 matches13 within13 the13 log13 records13 You13 can13 use13 Apache13 Lucene13 or13 Elascsearch13 query13 DSL13 syntax13

13

Page of 23 71 Page of 23 71

Saved13 searches13 You13 can13 save13 any13 custom13 search13 syntax13 and13 choice13 of13 me13 range13 for13 future13 ease13 of13 use13 Any13 relave13 or13 absolute13 me13 frame13 is13 saved13 as13 part13 of13 the13 search13 13

To13 save13 a13 search13

1 Set13 up13 and13 run13 the13 required13 syntax13 and13 me13 frame13

13

2 Click13 the13 Save13 Search13 icon13

13

3 Type13 in13 a13 name13 for13 the13 saved13 search13

Page of 24 71 Page of 24 71

13

7 To13 delete13 or13 update13 saved13 searches13 click13 13 manage13 saved13 searches13

13

Use13 the13 edit13 icons13 to13 make13 any13 changes13

Page of 26 71 Page of 26 71

13

Build13 simple13 Kibana13 histograms13 Kibana13 allows13 you13 to13 build13 charts13 and13 dashboards13 based13 on13 the13 search13 capability13 offered13 by13 the13 Discover13 page13 Saved13 searches13 can13 be13 used13 as13 the13 basis13 for13 charts13 and13 dashboards13 built13 using13 those13 charts13 To13 build13 a13 simple13 histogram13

1 Select13 the13 Visualize13 page13

13

2 Select13 Ver0cal13 Bar13 Chart13

13

3 To13 use13 a13 saved13 search13 click13 From13 a13 saved13 search13

Page of 27 71 Page of 27 71

13

4 Select13 a13 previously13 saved13 search13

13

5 Select13 to13 edit13 the13 x-shy‐axis

6 Edit13 the13 fields13 with13 informaon13 about13 the13 aggregaon13 field13 interval13 and13 custom13 label13

Page of 28 71 Page of 28 71

13

Page of 29 71 Page of 29 71

7 Click13 the13 green13 arrow13 The13 histogram13 is13 created13 and13 run13 based13 on13 the13 saved13 search

13

8 To13 save13 the13 histogram13 click13 the13 Save13 Visualiza0on13 icon13 and13 add13 a13 tle13

13

13

9 Click13 Save13

10 To13 view13 the13 visualisaon13 click13 the13 Dashboard13 page13

Page of 30 71 Page of 30 71

13

11 Click13 the13 Add13 Visualiza0on13 icon13

13

12 Select13 the13 visiualizaon13 that13 you13 just13 created13

13

Page of 31 71 Page of 31 71

13

13 To13 save13 the13 dashboard13 click13 the13 Save13 Dashboard13 icon13

Page of 32 71 Page of 32 71

13

14 Add13 a13 tle13 and13 select13 whether13 to13 store13 the13 selected13 me13 frame13 with13 the13 dashboard13

13

15 Click13 Save13

16 You13 can13 edit13 the13 shape13 and13 number13 of13 charts13 and13 save13 the13 dashboard13

17 To13 re-shy‐run13 the13 dashboard13 select13 the13 Dashboard13 page13 and13 click13 the13 Load13 Saved13 Dashboard13 icon13

Page of 33 71 Page of 33 71

13

18 Select13 the13 saved13 dashboard13 from13 the13 list13

13

19 The13 dashboard13 runs13

13

Page of 34 71 Page of 34 71

Page of 35 71 Page of 35 71

Aler0ng13 with13 ELK13 ELK13 examines13 the13 log13 records13 in13 process13 to13 send13 alerts13 to13 third-shy‐party13 alerng13 event13 management13 and13 social13 media13 applicaons13 As13 shown13 previously13 in13 the13 ELK13 architecture13 the13 ELK13 strategy13 for13 alerng13 is13 to13 send13 alerts13 from13 the13 Logstash13 component13 directly13 Therefore13 log13 records13 and13 alerts13 are13 sent13 to13 both13 Elascsearch13 and13 third-shy‐party13 tools13 at13 the13 same13 instant13 using13 the13 output13 secon13 of13 the13 Logstash13 configuraon13 Any13 alerts13 are13 generated13 during13 the13 filter13 secon13 processing13 of13 the13 Logstash13 configuraon13

Create13 alerts13 in13 Logstash13 The13 Logstash13 filter13 secon13 is13 used13 to13 trigger13 alerts13 based13 on13 the13 log13 records13 being13 processed13 For13 example13 log13 records13 can13 be13 checked13 for13 errors13 warnings13 crashed13 or13 stopped13 messages13 and13 generate13 an13 alert13

Here13 is13 an13 example13 Logstash13 filter13 code13

13 if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

The13 specific13 tags13 are13 set13 against13 the13 log13 records13 to13 trigger13 the13 alert13 later13 in13 the13 configuraon13 in13 the13 output13 secon13

Control13 the13 number13 of13 generated13 alerts13 using13 the13 Logstash13 throcle13 plugin13 so13 that13 either13 a13 lot13 of13 instances13 in13 a13 fixed13 me13 period13 are13 required13 before13 an13 alert13 is13 sent13 or13 a13 specific13 number13 of13 instances13 triggers13 the13 alert13

Page of 36 71 Page of 36 71

Edit13 the13 plugin13 parameters13 to13 meet13 system13 requirements13

13 13 13 13 13 if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 10

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle13

after_count =gt 20

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 30

period =gt 1200

key =gt servicename_alarm_warning

Page of 37 71 Page of 37 71

add_tag =gt servicename_throttled

13 13 13 13 13 13 13 13 13

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 40

period =gt 1500

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

Enrich13 alerts13 with13 message13 detail13 Some13 of13 the13 third-shy‐party13 integraons13 benefit13 from13 having13 components13 of13 the13 raw13 log13 record13 sent13 in13 addion13 to13 the13 raw13 record13 or13 fields13 in13 the13 record13 edited13 transformed13 or13 translated13

Grok13 Logstash13 allows13 the13 log13 record13 to13 be13 broken13 up13 into13 logical13 secons13 and13 each13 secon13 stored13 in13 a13 new13 parameter13 using13 the13 grok13 parameter13 in13 the13 filter13 secon13 Here13 is13 an13 example13 grok13

13 grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

To13 build13 the13 grok13 statement13 Logstash13 offers13 an13 online13 tool13 at13 hcpsgrokdebugherokuappcom13

Translate13 If13 specific13 fields13 in13 the13 log13 record13 are13 coded13 (for13 example13 the13 Bluemix13 app13 name13 is13 oaen13 not13 sent13 in13 the13 syslog13 messages13 instead13 the13 GUID13 is13 sent13 which13 is13 not13 useful)13 then13 the13 filter13 secon13 translate13 plugin13 can13 be13 used13 to13 add13 new13 fields13 by13 holding13 the13 decoded13 versions13 of13 those13 values13

13 translate

field =gt data1

destination =gt data1

override =gt true

Page of 38 71 Page of 38 71

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

dc50f6e4-c052-436a-bf27-95ba392c4228 monverifyservicemanagement

0c677e82-5bfc-4e38-9155-e416249e3ee6 ms-catalog-postpyretic-pomiculture

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

However13 the13 translate13 plugin13 is13 not13 included13 with13 the13 Logstash13 installaon13 In13 the13 next13 secon13 I13 will13 13 describe13 how13 to13 add13 new13 plugins13 to13 the13 Logstash13 instance13

Add13 custom13 plugins13 to13 Logstash13 As13 an13 open13 source13 tool13 Logstash13 encourages13 users13 to13 develop13 content13 for13 all13 users13 to13 share13 Some13 of13 the13 plugins13 that13 were13 created13 by13 our13 community13 include13 a13 translaon13 plugin13 an13 output13 Slack13 plugin13 and13 an13 output13 syslog13 plugin13 Letrsquos13 look13 at13 how13 to13 add13 these13 plugins13 to13 your13 environment13

Filter13 translate13 plugin13 To13 add13 the13 translate13 plugin13

1 Download13 the13 plugin13

2 Copy13 the13 plugin13 file13 logstash-shy‐filter-shy‐translate-shy‐214gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-filter-translate 214

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Output13 Slack13 plugin13 To13 add13 the13 Slack13 plugin13

Page of 39 71 Page of 39 71

1 Download13 the13 plugin13 13

2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐slack-shy‐014gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-output-slack 014

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Output13 syslog13 plugin13 To13 add13 the13 syslog13 plugin13

1 Download13 the13 plugin13 13

2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐syslog-shy‐214gem13 to13 the13 ELK13 Virtual13 Server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-output-syslog 214

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Integrate13 Logstash13 with13 Slack13 To13 send13 generated13 alerts13 to13 Slack13 complete13 these13 steps13

1 Create13 a13 Slack13 integraon13 13

2 Use13 the13 Slack13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 Slack13 integraon13

113 Login13 to13 Slack13 and13 go13 to13 the13 Apps13 page13 13

Page of 40 71 Page of 40 71

13

313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13

13

A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13

13

Page of 42 71 Page of 42 71

The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13

13

The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13

13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

slack

url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5

channel =gt cloud-operations-mon

username =gt nick_cawood

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13

Page of 43 71 Page of 43 71

Page of 44 71 Page of 44 71

In13 Slack13 the13 alerts13 look13 like13 this13

13

Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13

1 Create13 a13 PagerDuty13 integraon13 13

2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13

1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13

13

2 Click13 Add13 New13 Service13

13

313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13

Page of 45 71 Page of 45 71

13

413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13

513 Click13 Add13 Service13

13

The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13

Page of 46 71 Page of 46 71

13

The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt be7eb1ea9c224018923e818d3e891d0b

incident_key =gt logstashhosttype

Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13

Page of 47 71 Page of 47 71

13

Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13

1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13

2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13

3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13

4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13

13 13 http

url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo

http_method =gt post

format =gt json

For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13

13 13 syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

Page of 48 71 Page of 48 71

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13

13

The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13

if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2

This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13

Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13

Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13

Logstash13 severity Shown13 in13 Omnibus13 as

alert Indeterminate

crical Warning

debug 7

emergency Clear

error Minor

informaonal 6

noce Crical

warning Major

Page of 49 71 Page of 49 71

13

Page of 50 71 Page of 50 71

Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13

input

tcp

port =gt 5000

type =gt syslog

filter

mutate

add_field =gt format =gt cf

grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

translate

field =gt data1

destination =gt data1

override =gt true

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

=== Check Alarm Criteria ===

Standard Key Words

if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

=== Alarm Exclusions ===

if [message] =~ (i)sometexthere

mutate remove_tag =gt servicename_alarm

Throttle alarms so we do not hit PD rate limits

Page of 52 71 Page of 52 71

if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 5

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle

after_count =gt 5

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 5

period =gt 1200

key =gt servicename_alarm_warning

add_tag =gt servicename_throttled

Page of 53 71 Page of 53 71

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

output

elasticsearch

hosts =gt localhost

stdout codec =gt rubydebug

if servicename_alarm_err in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

Page of 54 71 Page of 54 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 55 71 Page of 55 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

Page of 56 71 Page of 56 71

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert err message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_error in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

Page of 57 71 Page of 57 71

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 58 71 Page of 58 71

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 59 71 Page of 59 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert error message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

Page of 60 71 Page of 60 71

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

Page of 61 71 Page of 61 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 62 71 Page of 62 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt x

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert fatal message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

Page of 63 71 Page of 63 71

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

Page of 64 71 Page of 64 71

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 65 71 Page of 65 71

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt heavy_exclamation_mark

Page of 66 71 Page of 66 71

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert WARNING message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 67 71 Page of 67 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

Page of 68 71 Page of 68 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 69 71 Page of 69 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

Page of 70 71 Page of 70 71

Page of 71 71 Page of 71 71

Page 18: How to integrate the Elasticsearch Logstash Kibana (ELK ... · How to integrate the Elasticsearch Logstash Kibana (ELK) log analytics stack into IBM Bluemix Nick Cawood IBM Cloud

Set the path to where you would like the process id file to be created

pidfile varrunkibanapid

If you would like to send the log output to a file you can set the path below

loggingdest stdout

Set this to true to suppress all logging output

loggingsilent false

Set this to true to suppress all logging output except for error messages

loggingquiet false

Set this to true to log all events including system usage information and all requests

loggingverbose false

Kibana13 is13 now13 installed13

Start13 Kibana13 To13 start13 Kibana13 go13 to13 the13 bin13 directory13 and13 run13 the13 kibana13 executable13

[rootelkforcase]$

[rootelkforcase]$ cd homekibanakibana-451-linux-x64bin

[rootelkforcase]$ kibana amp

[1] 22283

[rootelkforcase] log [095900542] [info][status][pluginkibana] Status changed from uninitialized to green - Ready

log [095900600] [info][status][pluginelasticsearch] Status changed from uninitialized to yellow - Waiting for Elasticsearch

log [095900689] [info][status][pluginkbn_vislib_vis_types] Status changed from uninitialized to green - Ready

log [095900717] [info][status][pluginmarkdown_vis] Status changed from uninitialized to green - Ready

log [095900721] [info][status][pluginmetric_vis] Status changed from uninitialized to green - Ready

log [095900814] [info][status][pluginspyModes] Status changed from uninitialized to green - Ready

log [095900934] [info][status][pluginstatusPage] Status changed from uninitialized to green - Ready

Page of 18 71 Page of 18 71

log [095900941] [info][status][plugintable_vis] Status changed from uninitialized to green - Ready

log [095901067] [info][listening] Server running at http00005601

log [095906083] [info][status][pluginelasticsearch] Status changed from yellow to yellow - No existing Kibana index found

log [095908942] [info][status][pluginelasticsearch] Status changed from yellow to green - Kibana index ready

Kibana13 is13 now13 started13

Broadcast13 Bluemix13 app13 log13 records13 Bluemix13 apps13 (which13 are13 shown13 in13 the13 US13 South13 Bluemix13 region13 of13 the13 System13 Architecture13 diagram)13 will13 produce13 logs13 in13 a13 syslog13 rfc542413 format13 using13 a13 Bluemix13 tool13 called13 Loggregator13

For13 Logstash13 to13 receive13 these13 logs13 each13 Bluemix13 app13 needs13 to13 broadcast13 these13 logs13 To13 do13 that13 run13 the13 following13 commands13 on13 each13 Bluemix13 app13 using13 the13 public13 IP13 address13 of13 the13 ELK13 virtual13 server13 the13 ELK13 tag13 to13 trace13 the13 process13 and13 insert13 the13 name13 of13 each13 app13 in13 turn13

cf cups elk -l syslogltpublicIPaddress5000 cf bind-service ltmyappgt elk

cf restage ltmyappgt

Install13 Logstash13 To13 install13 Logstash13 as13 root13 user13 create13 a13 directory13 for13 the13 installaon13 and13 untar13 the13 installaon13 file13 there13

13 13 [rootelkforcase]$ cd home

[rootelkforcase]$ mkdir logstash

[rootelkforcase]$ cp homeibmcloudlogstash-233targz homelogstash

[rootelkforcase]$ cd logstash

[rootelkforcase]$ gunzip logstash-233targz

[rootelkforcase]$ untar logstash-233tar

Write13 Logstash13 configura0on13 Logstash13 requires13 a13 configuraon13 file13 to13 be13 wricen13 to13 receive13 transform13 and13 output13 the13 log13 records13 to13 Elascsearch13 This13 secon13 explains13 the13 basic13 requirements13 of13 processing13 Bluemix13 app13 log13 records13 into13 Elascsearch13 (more13 complex13 configuraons13 are13 detailed13 later13 in13 this13 document13 for13 the13 integraons13 with13 third-shy‐party13 tools)13

Create13 a13 conf13 file13 as13 root13 user13

[rootelkforcase]$ cd homelogstashlogstash-233

[rootelkforcase]$ mkdir config

Page of 19 71 Page of 19 71

[rootelkforcase]$ cd config

[rootelkforcase]$ touch loggregatorconf

Edit13 the13 loggregator13 file13 with13 the13 following13 contents13

13 Input13 secon13 defines13 which13 log13 records13 to13 receive13

13 port13 and13 type13 reflect13 method13 selected13 for13 Cloud13 Foundry13 to13 broadcast13 the13 records13

input

tcp

port =gt 5000

type =gt syslog

13 Filter13 secon13 ndash13 transforms13 the13 data13

filter

Add a field to identify the records as Cloud Foundry based

mutate

add_field =gt format =gt cf

13 Output13 secon13 ndash13 sends13 the13 data13 to13 its13 desnaon(s)13

output

13 Use13 the13 elascsearch13 plugin13 to13 send13 the13 data13 to13 the13 elascsearch13 instance13

13 elasticsearch

hosts =gt localhost

stdout codec =gt rubydebug

13 End13 of13 Config13

Save13 the13 file13 Verify13 the13 format13 of13 the13 file13 is13 acceptable13 by13 running13 the13 following13 commands13

[rootelkforcase]$ cd homelogstashlogstash-233

[rootelkforcase]$ binlogstash agent --configtest -f configloggregatorconf

Page of 20 71 Page of 20 71

Configuraon13 is13 OK13

Start13 Logstash13 Aaer13 you13 have13 wricen13 and13 verified13 the13 Logstash13 configuraon13 you13 can13 start13 Logstash13 to13 process13 records13 from13 the13 Bluemix13 apps13 (via13 Cloud13 Foundry)13 Run13 the13 following13 as13 root13 user13

[rootelkforcase]$ cd homelogstashlogstash-233

[rootelkforcase]$ binlogstash agent -f configloggregatorconf

Logstash13 is13 now13 started13

Monitor13 Logstash13 processing13 Once13 Logstash13 is13 running13 and13 log13 records13 are13 being13 produced13 by13 the13 Bluemix13 apps13 you13 can13 monitor13 progress13 in13 the13 same13 window13 that13 was13 used13 to13 start13 Logstash13 Output13 is13 similar13 to13 the13 following13

13

The13 log13 records13 being13 broadcast13 from13 the13 Bluemix13 apps13 are13 now13 sent13 to13 Elascsearch13 from13 the13 Logstash13 output13 configuraon13 Records13 are13 sent13 in13 real13 me13 If13 Logstash13 is13 not13 running13 while13 log13 records13 are13 being13 produced13 then13 these13 records13 will13 not13 be13 processed13 by13 Logstash13 and13 will13 not13 be13 stored13 in13 Elascsearch13

Page of 21 71 Page of 21 71

Use13 ELK13 The13 log13 records13 from13 the13 Bluemix13 apps13 are13 now13 being13 stored13 and13 indexed13 by13 the13 Elascsearch13 component13 This13 data13 can13 be13 accessed13 via13 the13 Kibana13 applicaon13

Kibana13 Discover13 page13 Using13 a13 web13 browser13 either13 directly13 from13 a13 client13 or13 using13 the13 VNC13 client13 access13 Kibana13 using13 the13 publicIPaddress13 of13 the13 ELK13 virtual13 server13

hcpltpublicIPaddressgt560113

No13 userpassword13 credenals13 are13 required13

Kibana13 defaults13 to13 a13 search13 of13 all13 records13 ()13 in13 the13 last13 1513 minutes13 and13 returns13 the13 results13

13

More13 complex13 searches13 are13 possible13 by13 changing13 the13 me13 filter13 and13 supplying13 search13 queries13

Change13 the13 0me13 filter13 To13 change13 the13 me13 ranges13 click13 the13 clock13 symbol13 in13 the13 top-shy‐right13 of13 the13 Kibana13 Discover13 page13

13

You13 will13 see13 the13 following13 opons13

13

Page of 22 71 Page of 22 71

A13 variety13 of13 quick13 me13 ranges13 can13 be13 selected13 or13 you13 can13 switch13 to13 relave13 or13 absolute13 me13 ranges13

Quick13 me13 ranges13 allow13 switching13 between13 last13 X13 minutes13 to13 last13 Y13 hours13 or13 Z13 days13 etc13

Relave13 me13 ranges13 allow13 more13 specific13 versions13 of13 the13 Quick13 opons13 from13 specific13 minuteshoursdays13 to13 now13

13

Absolute13 me13 ranges13 specify13 the13 start13 and13 end13 mes13 for13 the13 searches13

13

Custom13 search13 queries13 You13 can13 use13 the13 text13 window13 of13 the13 Discover13 page13 to13 search13 for13 matches13 within13 the13 log13 records13 You13 can13 use13 Apache13 Lucene13 or13 Elascsearch13 query13 DSL13 syntax13

13

Page of 23 71 Page of 23 71

Saved13 searches13 You13 can13 save13 any13 custom13 search13 syntax13 and13 choice13 of13 me13 range13 for13 future13 ease13 of13 use13 Any13 relave13 or13 absolute13 me13 frame13 is13 saved13 as13 part13 of13 the13 search13 13

To13 save13 a13 search13

1 Set13 up13 and13 run13 the13 required13 syntax13 and13 me13 frame13

13

2 Click13 the13 Save13 Search13 icon13

13

3 Type13 in13 a13 name13 for13 the13 saved13 search13

Page of 24 71 Page of 24 71

13

7 To13 delete13 or13 update13 saved13 searches13 click13 13 manage13 saved13 searches13

13

Use13 the13 edit13 icons13 to13 make13 any13 changes13

Page of 26 71 Page of 26 71

13

Build13 simple13 Kibana13 histograms13 Kibana13 allows13 you13 to13 build13 charts13 and13 dashboards13 based13 on13 the13 search13 capability13 offered13 by13 the13 Discover13 page13 Saved13 searches13 can13 be13 used13 as13 the13 basis13 for13 charts13 and13 dashboards13 built13 using13 those13 charts13 To13 build13 a13 simple13 histogram13

1 Select13 the13 Visualize13 page13

13

2 Select13 Ver0cal13 Bar13 Chart13

13

3 To13 use13 a13 saved13 search13 click13 From13 a13 saved13 search13

Page of 27 71 Page of 27 71

13

4 Select13 a13 previously13 saved13 search13

13

5 Select13 to13 edit13 the13 x-shy‐axis

6 Edit13 the13 fields13 with13 informaon13 about13 the13 aggregaon13 field13 interval13 and13 custom13 label13

Page of 28 71 Page of 28 71

13

Page of 29 71 Page of 29 71

7 Click13 the13 green13 arrow13 The13 histogram13 is13 created13 and13 run13 based13 on13 the13 saved13 search

13

8 To13 save13 the13 histogram13 click13 the13 Save13 Visualiza0on13 icon13 and13 add13 a13 tle13

13

13

9 Click13 Save13

10 To13 view13 the13 visualisaon13 click13 the13 Dashboard13 page13

Page of 30 71 Page of 30 71

13

11 Click13 the13 Add13 Visualiza0on13 icon13

13

12 Select13 the13 visiualizaon13 that13 you13 just13 created13

13

Page of 31 71 Page of 31 71

13

13 To13 save13 the13 dashboard13 click13 the13 Save13 Dashboard13 icon13

Page of 32 71 Page of 32 71

13

14 Add13 a13 tle13 and13 select13 whether13 to13 store13 the13 selected13 me13 frame13 with13 the13 dashboard13

13

15 Click13 Save13

16 You13 can13 edit13 the13 shape13 and13 number13 of13 charts13 and13 save13 the13 dashboard13

17 To13 re-shy‐run13 the13 dashboard13 select13 the13 Dashboard13 page13 and13 click13 the13 Load13 Saved13 Dashboard13 icon13

Page of 33 71 Page of 33 71

13

18 Select13 the13 saved13 dashboard13 from13 the13 list13

13

19 The13 dashboard13 runs13

13

Page of 34 71 Page of 34 71

Page of 35 71 Page of 35 71

Aler0ng13 with13 ELK13 ELK13 examines13 the13 log13 records13 in13 process13 to13 send13 alerts13 to13 third-shy‐party13 alerng13 event13 management13 and13 social13 media13 applicaons13 As13 shown13 previously13 in13 the13 ELK13 architecture13 the13 ELK13 strategy13 for13 alerng13 is13 to13 send13 alerts13 from13 the13 Logstash13 component13 directly13 Therefore13 log13 records13 and13 alerts13 are13 sent13 to13 both13 Elascsearch13 and13 third-shy‐party13 tools13 at13 the13 same13 instant13 using13 the13 output13 secon13 of13 the13 Logstash13 configuraon13 Any13 alerts13 are13 generated13 during13 the13 filter13 secon13 processing13 of13 the13 Logstash13 configuraon13

Create13 alerts13 in13 Logstash13 The13 Logstash13 filter13 secon13 is13 used13 to13 trigger13 alerts13 based13 on13 the13 log13 records13 being13 processed13 For13 example13 log13 records13 can13 be13 checked13 for13 errors13 warnings13 crashed13 or13 stopped13 messages13 and13 generate13 an13 alert13

Here13 is13 an13 example13 Logstash13 filter13 code13

13 if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

The13 specific13 tags13 are13 set13 against13 the13 log13 records13 to13 trigger13 the13 alert13 later13 in13 the13 configuraon13 in13 the13 output13 secon13

Control13 the13 number13 of13 generated13 alerts13 using13 the13 Logstash13 throcle13 plugin13 so13 that13 either13 a13 lot13 of13 instances13 in13 a13 fixed13 me13 period13 are13 required13 before13 an13 alert13 is13 sent13 or13 a13 specific13 number13 of13 instances13 triggers13 the13 alert13

Page of 36 71 Page of 36 71

Edit13 the13 plugin13 parameters13 to13 meet13 system13 requirements13

13 13 13 13 13 if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 10

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle13

after_count =gt 20

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 30

period =gt 1200

key =gt servicename_alarm_warning

Page of 37 71 Page of 37 71

add_tag =gt servicename_throttled

13 13 13 13 13 13 13 13 13

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 40

period =gt 1500

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

Enrich13 alerts13 with13 message13 detail13 Some13 of13 the13 third-shy‐party13 integraons13 benefit13 from13 having13 components13 of13 the13 raw13 log13 record13 sent13 in13 addion13 to13 the13 raw13 record13 or13 fields13 in13 the13 record13 edited13 transformed13 or13 translated13

Grok13 Logstash13 allows13 the13 log13 record13 to13 be13 broken13 up13 into13 logical13 secons13 and13 each13 secon13 stored13 in13 a13 new13 parameter13 using13 the13 grok13 parameter13 in13 the13 filter13 secon13 Here13 is13 an13 example13 grok13

13 grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

To13 build13 the13 grok13 statement13 Logstash13 offers13 an13 online13 tool13 at13 hcpsgrokdebugherokuappcom13

Translate13 If13 specific13 fields13 in13 the13 log13 record13 are13 coded13 (for13 example13 the13 Bluemix13 app13 name13 is13 oaen13 not13 sent13 in13 the13 syslog13 messages13 instead13 the13 GUID13 is13 sent13 which13 is13 not13 useful)13 then13 the13 filter13 secon13 translate13 plugin13 can13 be13 used13 to13 add13 new13 fields13 by13 holding13 the13 decoded13 versions13 of13 those13 values13

13 translate

field =gt data1

destination =gt data1

override =gt true

Page of 38 71 Page of 38 71

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

dc50f6e4-c052-436a-bf27-95ba392c4228 monverifyservicemanagement

0c677e82-5bfc-4e38-9155-e416249e3ee6 ms-catalog-postpyretic-pomiculture

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

However13 the13 translate13 plugin13 is13 not13 included13 with13 the13 Logstash13 installaon13 In13 the13 next13 secon13 I13 will13 13 describe13 how13 to13 add13 new13 plugins13 to13 the13 Logstash13 instance13

Add13 custom13 plugins13 to13 Logstash13 As13 an13 open13 source13 tool13 Logstash13 encourages13 users13 to13 develop13 content13 for13 all13 users13 to13 share13 Some13 of13 the13 plugins13 that13 were13 created13 by13 our13 community13 include13 a13 translaon13 plugin13 an13 output13 Slack13 plugin13 and13 an13 output13 syslog13 plugin13 Letrsquos13 look13 at13 how13 to13 add13 these13 plugins13 to13 your13 environment13

Filter13 translate13 plugin13 To13 add13 the13 translate13 plugin13

1 Download13 the13 plugin13

2 Copy13 the13 plugin13 file13 logstash-shy‐filter-shy‐translate-shy‐214gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-filter-translate 214

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Output13 Slack13 plugin13 To13 add13 the13 Slack13 plugin13

Page of 39 71 Page of 39 71

1 Download13 the13 plugin13 13

2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐slack-shy‐014gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-output-slack 014

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Output13 syslog13 plugin13 To13 add13 the13 syslog13 plugin13

1 Download13 the13 plugin13 13

2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐syslog-shy‐214gem13 to13 the13 ELK13 Virtual13 Server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-output-syslog 214

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Integrate13 Logstash13 with13 Slack13 To13 send13 generated13 alerts13 to13 Slack13 complete13 these13 steps13

1 Create13 a13 Slack13 integraon13 13

2 Use13 the13 Slack13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 Slack13 integraon13

113 Login13 to13 Slack13 and13 go13 to13 the13 Apps13 page13 13

Page of 40 71 Page of 40 71

13

313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13

13

A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13

13

Page of 42 71 Page of 42 71

The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13

13

The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13

13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

slack

url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5

channel =gt cloud-operations-mon

username =gt nick_cawood

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13

Page of 43 71 Page of 43 71

Page of 44 71 Page of 44 71

In13 Slack13 the13 alerts13 look13 like13 this13

13

Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13

1 Create13 a13 PagerDuty13 integraon13 13

2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13

1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13

13

2 Click13 Add13 New13 Service13

13

313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13

Page of 45 71 Page of 45 71

13

413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13

513 Click13 Add13 Service13

13

The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13

Page of 46 71 Page of 46 71

13

The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt be7eb1ea9c224018923e818d3e891d0b

incident_key =gt logstashhosttype

Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13

Page of 47 71 Page of 47 71

13

Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13

1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13

2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13

3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13

4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13

13 13 http

url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo

http_method =gt post

format =gt json

For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13

13 13 syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

Page of 48 71 Page of 48 71

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13

13

The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13

if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2

This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13

Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13

Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13

Logstash13 severity Shown13 in13 Omnibus13 as

alert Indeterminate

crical Warning

debug 7

emergency Clear

error Minor

informaonal 6

noce Crical

warning Major

Page of 49 71 Page of 49 71

13

Page of 50 71 Page of 50 71

Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13

input

tcp

port =gt 5000

type =gt syslog

filter

mutate

add_field =gt format =gt cf

grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

translate

field =gt data1

destination =gt data1

override =gt true

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

=== Check Alarm Criteria ===

Standard Key Words

if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

=== Alarm Exclusions ===

if [message] =~ (i)sometexthere

mutate remove_tag =gt servicename_alarm

Throttle alarms so we do not hit PD rate limits

Page of 52 71 Page of 52 71

if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 5

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle

after_count =gt 5

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 5

period =gt 1200

key =gt servicename_alarm_warning

add_tag =gt servicename_throttled

Page of 53 71 Page of 53 71

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

output

elasticsearch

hosts =gt localhost

stdout codec =gt rubydebug

if servicename_alarm_err in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

Page of 54 71 Page of 54 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 55 71 Page of 55 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

Page of 56 71 Page of 56 71

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert err message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_error in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

Page of 57 71 Page of 57 71

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 58 71 Page of 58 71

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 59 71 Page of 59 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert error message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

Page of 60 71 Page of 60 71

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

Page of 61 71 Page of 61 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 62 71 Page of 62 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt x

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert fatal message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

Page of 63 71 Page of 63 71

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

Page of 64 71 Page of 64 71

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 65 71 Page of 65 71

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt heavy_exclamation_mark

Page of 66 71 Page of 66 71

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert WARNING message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 67 71 Page of 67 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

Page of 68 71 Page of 68 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 69 71 Page of 69 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

Page of 70 71 Page of 70 71

Page of 71 71 Page of 71 71

Page 19: How to integrate the Elasticsearch Logstash Kibana (ELK ... · How to integrate the Elasticsearch Logstash Kibana (ELK) log analytics stack into IBM Bluemix Nick Cawood IBM Cloud

log [095900941] [info][status][plugintable_vis] Status changed from uninitialized to green - Ready

log [095901067] [info][listening] Server running at http00005601

log [095906083] [info][status][pluginelasticsearch] Status changed from yellow to yellow - No existing Kibana index found

log [095908942] [info][status][pluginelasticsearch] Status changed from yellow to green - Kibana index ready

Kibana13 is13 now13 started13

Broadcast13 Bluemix13 app13 log13 records13 Bluemix13 apps13 (which13 are13 shown13 in13 the13 US13 South13 Bluemix13 region13 of13 the13 System13 Architecture13 diagram)13 will13 produce13 logs13 in13 a13 syslog13 rfc542413 format13 using13 a13 Bluemix13 tool13 called13 Loggregator13

For13 Logstash13 to13 receive13 these13 logs13 each13 Bluemix13 app13 needs13 to13 broadcast13 these13 logs13 To13 do13 that13 run13 the13 following13 commands13 on13 each13 Bluemix13 app13 using13 the13 public13 IP13 address13 of13 the13 ELK13 virtual13 server13 the13 ELK13 tag13 to13 trace13 the13 process13 and13 insert13 the13 name13 of13 each13 app13 in13 turn13

cf cups elk -l syslogltpublicIPaddress5000 cf bind-service ltmyappgt elk

cf restage ltmyappgt

Install13 Logstash13 To13 install13 Logstash13 as13 root13 user13 create13 a13 directory13 for13 the13 installaon13 and13 untar13 the13 installaon13 file13 there13

13 13 [rootelkforcase]$ cd home

[rootelkforcase]$ mkdir logstash

[rootelkforcase]$ cp homeibmcloudlogstash-233targz homelogstash

[rootelkforcase]$ cd logstash

[rootelkforcase]$ gunzip logstash-233targz

[rootelkforcase]$ untar logstash-233tar

Write13 Logstash13 configura0on13 Logstash13 requires13 a13 configuraon13 file13 to13 be13 wricen13 to13 receive13 transform13 and13 output13 the13 log13 records13 to13 Elascsearch13 This13 secon13 explains13 the13 basic13 requirements13 of13 processing13 Bluemix13 app13 log13 records13 into13 Elascsearch13 (more13 complex13 configuraons13 are13 detailed13 later13 in13 this13 document13 for13 the13 integraons13 with13 third-shy‐party13 tools)13

Create13 a13 conf13 file13 as13 root13 user13

[rootelkforcase]$ cd homelogstashlogstash-233

[rootelkforcase]$ mkdir config

Page of 19 71 Page of 19 71

[rootelkforcase]$ cd config

[rootelkforcase]$ touch loggregatorconf

Edit13 the13 loggregator13 file13 with13 the13 following13 contents13

13 Input13 secon13 defines13 which13 log13 records13 to13 receive13

13 port13 and13 type13 reflect13 method13 selected13 for13 Cloud13 Foundry13 to13 broadcast13 the13 records13

input

tcp

port =gt 5000

type =gt syslog

13 Filter13 secon13 ndash13 transforms13 the13 data13

filter

Add a field to identify the records as Cloud Foundry based

mutate

add_field =gt format =gt cf

13 Output13 secon13 ndash13 sends13 the13 data13 to13 its13 desnaon(s)13

output

13 Use13 the13 elascsearch13 plugin13 to13 send13 the13 data13 to13 the13 elascsearch13 instance13

13 elasticsearch

hosts =gt localhost

stdout codec =gt rubydebug

13 End13 of13 Config13

Save13 the13 file13 Verify13 the13 format13 of13 the13 file13 is13 acceptable13 by13 running13 the13 following13 commands13

[rootelkforcase]$ cd homelogstashlogstash-233

[rootelkforcase]$ binlogstash agent --configtest -f configloggregatorconf

Page of 20 71 Page of 20 71

Configuraon13 is13 OK13

Start13 Logstash13 Aaer13 you13 have13 wricen13 and13 verified13 the13 Logstash13 configuraon13 you13 can13 start13 Logstash13 to13 process13 records13 from13 the13 Bluemix13 apps13 (via13 Cloud13 Foundry)13 Run13 the13 following13 as13 root13 user13

[rootelkforcase]$ cd homelogstashlogstash-233

[rootelkforcase]$ binlogstash agent -f configloggregatorconf

Logstash13 is13 now13 started13

Monitor13 Logstash13 processing13 Once13 Logstash13 is13 running13 and13 log13 records13 are13 being13 produced13 by13 the13 Bluemix13 apps13 you13 can13 monitor13 progress13 in13 the13 same13 window13 that13 was13 used13 to13 start13 Logstash13 Output13 is13 similar13 to13 the13 following13

13

The13 log13 records13 being13 broadcast13 from13 the13 Bluemix13 apps13 are13 now13 sent13 to13 Elascsearch13 from13 the13 Logstash13 output13 configuraon13 Records13 are13 sent13 in13 real13 me13 If13 Logstash13 is13 not13 running13 while13 log13 records13 are13 being13 produced13 then13 these13 records13 will13 not13 be13 processed13 by13 Logstash13 and13 will13 not13 be13 stored13 in13 Elascsearch13

Page of 21 71 Page of 21 71

Use13 ELK13 The13 log13 records13 from13 the13 Bluemix13 apps13 are13 now13 being13 stored13 and13 indexed13 by13 the13 Elascsearch13 component13 This13 data13 can13 be13 accessed13 via13 the13 Kibana13 applicaon13

Kibana13 Discover13 page13 Using13 a13 web13 browser13 either13 directly13 from13 a13 client13 or13 using13 the13 VNC13 client13 access13 Kibana13 using13 the13 publicIPaddress13 of13 the13 ELK13 virtual13 server13

hcpltpublicIPaddressgt560113

No13 userpassword13 credenals13 are13 required13

Kibana13 defaults13 to13 a13 search13 of13 all13 records13 ()13 in13 the13 last13 1513 minutes13 and13 returns13 the13 results13

13

More13 complex13 searches13 are13 possible13 by13 changing13 the13 me13 filter13 and13 supplying13 search13 queries13

Change13 the13 0me13 filter13 To13 change13 the13 me13 ranges13 click13 the13 clock13 symbol13 in13 the13 top-shy‐right13 of13 the13 Kibana13 Discover13 page13

13

You13 will13 see13 the13 following13 opons13

13

Page of 22 71 Page of 22 71

A13 variety13 of13 quick13 me13 ranges13 can13 be13 selected13 or13 you13 can13 switch13 to13 relave13 or13 absolute13 me13 ranges13

Quick13 me13 ranges13 allow13 switching13 between13 last13 X13 minutes13 to13 last13 Y13 hours13 or13 Z13 days13 etc13

Relave13 me13 ranges13 allow13 more13 specific13 versions13 of13 the13 Quick13 opons13 from13 specific13 minuteshoursdays13 to13 now13

13

Absolute13 me13 ranges13 specify13 the13 start13 and13 end13 mes13 for13 the13 searches13

13

Custom13 search13 queries13 You13 can13 use13 the13 text13 window13 of13 the13 Discover13 page13 to13 search13 for13 matches13 within13 the13 log13 records13 You13 can13 use13 Apache13 Lucene13 or13 Elascsearch13 query13 DSL13 syntax13

13

Page of 23 71 Page of 23 71

Saved13 searches13 You13 can13 save13 any13 custom13 search13 syntax13 and13 choice13 of13 me13 range13 for13 future13 ease13 of13 use13 Any13 relave13 or13 absolute13 me13 frame13 is13 saved13 as13 part13 of13 the13 search13 13

To13 save13 a13 search13

1 Set13 up13 and13 run13 the13 required13 syntax13 and13 me13 frame13

13

2 Click13 the13 Save13 Search13 icon13

13

3 Type13 in13 a13 name13 for13 the13 saved13 search13

Page of 24 71 Page of 24 71

13

7 To13 delete13 or13 update13 saved13 searches13 click13 13 manage13 saved13 searches13

13

Use13 the13 edit13 icons13 to13 make13 any13 changes13

Page of 26 71 Page of 26 71

13

Build13 simple13 Kibana13 histograms13 Kibana13 allows13 you13 to13 build13 charts13 and13 dashboards13 based13 on13 the13 search13 capability13 offered13 by13 the13 Discover13 page13 Saved13 searches13 can13 be13 used13 as13 the13 basis13 for13 charts13 and13 dashboards13 built13 using13 those13 charts13 To13 build13 a13 simple13 histogram13

1 Select13 the13 Visualize13 page13

13

2 Select13 Ver0cal13 Bar13 Chart13

13

3 To13 use13 a13 saved13 search13 click13 From13 a13 saved13 search13

Page of 27 71 Page of 27 71

13

4 Select13 a13 previously13 saved13 search13

13

5 Select13 to13 edit13 the13 x-shy‐axis

6 Edit13 the13 fields13 with13 informaon13 about13 the13 aggregaon13 field13 interval13 and13 custom13 label13

Page of 28 71 Page of 28 71

13

Page of 29 71 Page of 29 71

7 Click13 the13 green13 arrow13 The13 histogram13 is13 created13 and13 run13 based13 on13 the13 saved13 search

13

8 To13 save13 the13 histogram13 click13 the13 Save13 Visualiza0on13 icon13 and13 add13 a13 tle13

13

13

9 Click13 Save13

10 To13 view13 the13 visualisaon13 click13 the13 Dashboard13 page13

Page of 30 71 Page of 30 71

13

11 Click13 the13 Add13 Visualiza0on13 icon13

13

12 Select13 the13 visiualizaon13 that13 you13 just13 created13

13

Page of 31 71 Page of 31 71

13

13 To13 save13 the13 dashboard13 click13 the13 Save13 Dashboard13 icon13

Page of 32 71 Page of 32 71

13

14 Add13 a13 tle13 and13 select13 whether13 to13 store13 the13 selected13 me13 frame13 with13 the13 dashboard13

13

15 Click13 Save13

16 You13 can13 edit13 the13 shape13 and13 number13 of13 charts13 and13 save13 the13 dashboard13

17 To13 re-shy‐run13 the13 dashboard13 select13 the13 Dashboard13 page13 and13 click13 the13 Load13 Saved13 Dashboard13 icon13

Page of 33 71 Page of 33 71

13

18 Select13 the13 saved13 dashboard13 from13 the13 list13

13

19 The13 dashboard13 runs13

13

Page of 34 71 Page of 34 71

Page of 35 71 Page of 35 71

Aler0ng13 with13 ELK13 ELK13 examines13 the13 log13 records13 in13 process13 to13 send13 alerts13 to13 third-shy‐party13 alerng13 event13 management13 and13 social13 media13 applicaons13 As13 shown13 previously13 in13 the13 ELK13 architecture13 the13 ELK13 strategy13 for13 alerng13 is13 to13 send13 alerts13 from13 the13 Logstash13 component13 directly13 Therefore13 log13 records13 and13 alerts13 are13 sent13 to13 both13 Elascsearch13 and13 third-shy‐party13 tools13 at13 the13 same13 instant13 using13 the13 output13 secon13 of13 the13 Logstash13 configuraon13 Any13 alerts13 are13 generated13 during13 the13 filter13 secon13 processing13 of13 the13 Logstash13 configuraon13

Create13 alerts13 in13 Logstash13 The13 Logstash13 filter13 secon13 is13 used13 to13 trigger13 alerts13 based13 on13 the13 log13 records13 being13 processed13 For13 example13 log13 records13 can13 be13 checked13 for13 errors13 warnings13 crashed13 or13 stopped13 messages13 and13 generate13 an13 alert13

Here13 is13 an13 example13 Logstash13 filter13 code13

13 if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

The13 specific13 tags13 are13 set13 against13 the13 log13 records13 to13 trigger13 the13 alert13 later13 in13 the13 configuraon13 in13 the13 output13 secon13

Control13 the13 number13 of13 generated13 alerts13 using13 the13 Logstash13 throcle13 plugin13 so13 that13 either13 a13 lot13 of13 instances13 in13 a13 fixed13 me13 period13 are13 required13 before13 an13 alert13 is13 sent13 or13 a13 specific13 number13 of13 instances13 triggers13 the13 alert13

Page of 36 71 Page of 36 71

Edit13 the13 plugin13 parameters13 to13 meet13 system13 requirements13

13 13 13 13 13 if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 10

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle13

after_count =gt 20

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 30

period =gt 1200

key =gt servicename_alarm_warning

Page of 37 71 Page of 37 71

add_tag =gt servicename_throttled

13 13 13 13 13 13 13 13 13

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 40

period =gt 1500

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

Enrich13 alerts13 with13 message13 detail13 Some13 of13 the13 third-shy‐party13 integraons13 benefit13 from13 having13 components13 of13 the13 raw13 log13 record13 sent13 in13 addion13 to13 the13 raw13 record13 or13 fields13 in13 the13 record13 edited13 transformed13 or13 translated13

Grok13 Logstash13 allows13 the13 log13 record13 to13 be13 broken13 up13 into13 logical13 secons13 and13 each13 secon13 stored13 in13 a13 new13 parameter13 using13 the13 grok13 parameter13 in13 the13 filter13 secon13 Here13 is13 an13 example13 grok13

13 grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

To13 build13 the13 grok13 statement13 Logstash13 offers13 an13 online13 tool13 at13 hcpsgrokdebugherokuappcom13

Translate13 If13 specific13 fields13 in13 the13 log13 record13 are13 coded13 (for13 example13 the13 Bluemix13 app13 name13 is13 oaen13 not13 sent13 in13 the13 syslog13 messages13 instead13 the13 GUID13 is13 sent13 which13 is13 not13 useful)13 then13 the13 filter13 secon13 translate13 plugin13 can13 be13 used13 to13 add13 new13 fields13 by13 holding13 the13 decoded13 versions13 of13 those13 values13

13 translate

field =gt data1

destination =gt data1

override =gt true

Page of 38 71 Page of 38 71

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

dc50f6e4-c052-436a-bf27-95ba392c4228 monverifyservicemanagement

0c677e82-5bfc-4e38-9155-e416249e3ee6 ms-catalog-postpyretic-pomiculture

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

However13 the13 translate13 plugin13 is13 not13 included13 with13 the13 Logstash13 installaon13 In13 the13 next13 secon13 I13 will13 13 describe13 how13 to13 add13 new13 plugins13 to13 the13 Logstash13 instance13

Add13 custom13 plugins13 to13 Logstash13 As13 an13 open13 source13 tool13 Logstash13 encourages13 users13 to13 develop13 content13 for13 all13 users13 to13 share13 Some13 of13 the13 plugins13 that13 were13 created13 by13 our13 community13 include13 a13 translaon13 plugin13 an13 output13 Slack13 plugin13 and13 an13 output13 syslog13 plugin13 Letrsquos13 look13 at13 how13 to13 add13 these13 plugins13 to13 your13 environment13

Filter13 translate13 plugin13 To13 add13 the13 translate13 plugin13

1 Download13 the13 plugin13

2 Copy13 the13 plugin13 file13 logstash-shy‐filter-shy‐translate-shy‐214gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-filter-translate 214

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Output13 Slack13 plugin13 To13 add13 the13 Slack13 plugin13

Page of 39 71 Page of 39 71

1 Download13 the13 plugin13 13

2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐slack-shy‐014gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-output-slack 014

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Output13 syslog13 plugin13 To13 add13 the13 syslog13 plugin13

1 Download13 the13 plugin13 13

2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐syslog-shy‐214gem13 to13 the13 ELK13 Virtual13 Server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-output-syslog 214

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Integrate13 Logstash13 with13 Slack13 To13 send13 generated13 alerts13 to13 Slack13 complete13 these13 steps13

1 Create13 a13 Slack13 integraon13 13

2 Use13 the13 Slack13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 Slack13 integraon13

113 Login13 to13 Slack13 and13 go13 to13 the13 Apps13 page13 13

Page of 40 71 Page of 40 71

13

313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13

13

A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13

13

Page of 42 71 Page of 42 71

The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13

13

The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13

13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

slack

url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5

channel =gt cloud-operations-mon

username =gt nick_cawood

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13

Page of 43 71 Page of 43 71

Page of 44 71 Page of 44 71

In13 Slack13 the13 alerts13 look13 like13 this13

13

Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13

1 Create13 a13 PagerDuty13 integraon13 13

2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13

1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13

13

2 Click13 Add13 New13 Service13

13

313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13

Page of 45 71 Page of 45 71

13

413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13

513 Click13 Add13 Service13

13

The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13

Page of 46 71 Page of 46 71

13

The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt be7eb1ea9c224018923e818d3e891d0b

incident_key =gt logstashhosttype

Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13

Page of 47 71 Page of 47 71

13

Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13

1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13

2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13

3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13

4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13

13 13 http

url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo

http_method =gt post

format =gt json

For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13

13 13 syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

Page of 48 71 Page of 48 71

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13

13

The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13

if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2

This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13

Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13

Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13

Logstash13 severity Shown13 in13 Omnibus13 as

alert Indeterminate

crical Warning

debug 7

emergency Clear

error Minor

informaonal 6

noce Crical

warning Major

Page of 49 71 Page of 49 71

13

Page of 50 71 Page of 50 71

Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13

input

tcp

port =gt 5000

type =gt syslog

filter

mutate

add_field =gt format =gt cf

grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

translate

field =gt data1

destination =gt data1

override =gt true

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

=== Check Alarm Criteria ===

Standard Key Words

if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

=== Alarm Exclusions ===

if [message] =~ (i)sometexthere

mutate remove_tag =gt servicename_alarm

Throttle alarms so we do not hit PD rate limits

Page of 52 71 Page of 52 71

if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 5

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle

after_count =gt 5

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 5

period =gt 1200

key =gt servicename_alarm_warning

add_tag =gt servicename_throttled

Page of 53 71 Page of 53 71

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

output

elasticsearch

hosts =gt localhost

stdout codec =gt rubydebug

if servicename_alarm_err in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

Page of 54 71 Page of 54 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 55 71 Page of 55 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

Page of 56 71 Page of 56 71

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert err message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_error in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

Page of 57 71 Page of 57 71

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 58 71 Page of 58 71

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 59 71 Page of 59 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert error message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

Page of 60 71 Page of 60 71

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

Page of 61 71 Page of 61 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 62 71 Page of 62 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt x

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert fatal message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

Page of 63 71 Page of 63 71

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

Page of 64 71 Page of 64 71

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 65 71 Page of 65 71

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt heavy_exclamation_mark

Page of 66 71 Page of 66 71

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert WARNING message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 67 71 Page of 67 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

Page of 68 71 Page of 68 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 69 71 Page of 69 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

Page of 70 71 Page of 70 71

Page of 71 71 Page of 71 71

Page 20: How to integrate the Elasticsearch Logstash Kibana (ELK ... · How to integrate the Elasticsearch Logstash Kibana (ELK) log analytics stack into IBM Bluemix Nick Cawood IBM Cloud

[rootelkforcase]$ cd config

[rootelkforcase]$ touch loggregatorconf

Edit13 the13 loggregator13 file13 with13 the13 following13 contents13

13 Input13 secon13 defines13 which13 log13 records13 to13 receive13

13 port13 and13 type13 reflect13 method13 selected13 for13 Cloud13 Foundry13 to13 broadcast13 the13 records13

input

tcp

port =gt 5000

type =gt syslog

13 Filter13 secon13 ndash13 transforms13 the13 data13

filter

Add a field to identify the records as Cloud Foundry based

mutate

add_field =gt format =gt cf

13 Output13 secon13 ndash13 sends13 the13 data13 to13 its13 desnaon(s)13

output

13 Use13 the13 elascsearch13 plugin13 to13 send13 the13 data13 to13 the13 elascsearch13 instance13

13 elasticsearch

hosts =gt localhost

stdout codec =gt rubydebug

13 End13 of13 Config13

Save13 the13 file13 Verify13 the13 format13 of13 the13 file13 is13 acceptable13 by13 running13 the13 following13 commands13

[rootelkforcase]$ cd homelogstashlogstash-233

[rootelkforcase]$ binlogstash agent --configtest -f configloggregatorconf

Page of 20 71 Page of 20 71

Configuraon13 is13 OK13

Start13 Logstash13 Aaer13 you13 have13 wricen13 and13 verified13 the13 Logstash13 configuraon13 you13 can13 start13 Logstash13 to13 process13 records13 from13 the13 Bluemix13 apps13 (via13 Cloud13 Foundry)13 Run13 the13 following13 as13 root13 user13

[rootelkforcase]$ cd homelogstashlogstash-233

[rootelkforcase]$ binlogstash agent -f configloggregatorconf

Logstash13 is13 now13 started13

Monitor13 Logstash13 processing13 Once13 Logstash13 is13 running13 and13 log13 records13 are13 being13 produced13 by13 the13 Bluemix13 apps13 you13 can13 monitor13 progress13 in13 the13 same13 window13 that13 was13 used13 to13 start13 Logstash13 Output13 is13 similar13 to13 the13 following13

13

The13 log13 records13 being13 broadcast13 from13 the13 Bluemix13 apps13 are13 now13 sent13 to13 Elascsearch13 from13 the13 Logstash13 output13 configuraon13 Records13 are13 sent13 in13 real13 me13 If13 Logstash13 is13 not13 running13 while13 log13 records13 are13 being13 produced13 then13 these13 records13 will13 not13 be13 processed13 by13 Logstash13 and13 will13 not13 be13 stored13 in13 Elascsearch13

Page of 21 71 Page of 21 71

Use13 ELK13 The13 log13 records13 from13 the13 Bluemix13 apps13 are13 now13 being13 stored13 and13 indexed13 by13 the13 Elascsearch13 component13 This13 data13 can13 be13 accessed13 via13 the13 Kibana13 applicaon13

Kibana13 Discover13 page13 Using13 a13 web13 browser13 either13 directly13 from13 a13 client13 or13 using13 the13 VNC13 client13 access13 Kibana13 using13 the13 publicIPaddress13 of13 the13 ELK13 virtual13 server13

hcpltpublicIPaddressgt560113

No13 userpassword13 credenals13 are13 required13

Kibana13 defaults13 to13 a13 search13 of13 all13 records13 ()13 in13 the13 last13 1513 minutes13 and13 returns13 the13 results13

13

More13 complex13 searches13 are13 possible13 by13 changing13 the13 me13 filter13 and13 supplying13 search13 queries13

Change13 the13 0me13 filter13 To13 change13 the13 me13 ranges13 click13 the13 clock13 symbol13 in13 the13 top-shy‐right13 of13 the13 Kibana13 Discover13 page13

13

You13 will13 see13 the13 following13 opons13

13

Page of 22 71 Page of 22 71

A13 variety13 of13 quick13 me13 ranges13 can13 be13 selected13 or13 you13 can13 switch13 to13 relave13 or13 absolute13 me13 ranges13

Quick13 me13 ranges13 allow13 switching13 between13 last13 X13 minutes13 to13 last13 Y13 hours13 or13 Z13 days13 etc13

Relave13 me13 ranges13 allow13 more13 specific13 versions13 of13 the13 Quick13 opons13 from13 specific13 minuteshoursdays13 to13 now13

13

Absolute13 me13 ranges13 specify13 the13 start13 and13 end13 mes13 for13 the13 searches13

13

Custom13 search13 queries13 You13 can13 use13 the13 text13 window13 of13 the13 Discover13 page13 to13 search13 for13 matches13 within13 the13 log13 records13 You13 can13 use13 Apache13 Lucene13 or13 Elascsearch13 query13 DSL13 syntax13

13

Page of 23 71 Page of 23 71

Saved13 searches13 You13 can13 save13 any13 custom13 search13 syntax13 and13 choice13 of13 me13 range13 for13 future13 ease13 of13 use13 Any13 relave13 or13 absolute13 me13 frame13 is13 saved13 as13 part13 of13 the13 search13 13

To13 save13 a13 search13

1 Set13 up13 and13 run13 the13 required13 syntax13 and13 me13 frame13

13

2 Click13 the13 Save13 Search13 icon13

13

3 Type13 in13 a13 name13 for13 the13 saved13 search13

Page of 24 71 Page of 24 71

13

7 To13 delete13 or13 update13 saved13 searches13 click13 13 manage13 saved13 searches13

13

Use13 the13 edit13 icons13 to13 make13 any13 changes13

Page of 26 71 Page of 26 71

13

Build13 simple13 Kibana13 histograms13 Kibana13 allows13 you13 to13 build13 charts13 and13 dashboards13 based13 on13 the13 search13 capability13 offered13 by13 the13 Discover13 page13 Saved13 searches13 can13 be13 used13 as13 the13 basis13 for13 charts13 and13 dashboards13 built13 using13 those13 charts13 To13 build13 a13 simple13 histogram13

1 Select13 the13 Visualize13 page13

13

2 Select13 Ver0cal13 Bar13 Chart13

13

3 To13 use13 a13 saved13 search13 click13 From13 a13 saved13 search13

Page of 27 71 Page of 27 71

13

4 Select13 a13 previously13 saved13 search13

13

5 Select13 to13 edit13 the13 x-shy‐axis

6 Edit13 the13 fields13 with13 informaon13 about13 the13 aggregaon13 field13 interval13 and13 custom13 label13

Page of 28 71 Page of 28 71

13

Page of 29 71 Page of 29 71

7 Click13 the13 green13 arrow13 The13 histogram13 is13 created13 and13 run13 based13 on13 the13 saved13 search

13

8 To13 save13 the13 histogram13 click13 the13 Save13 Visualiza0on13 icon13 and13 add13 a13 tle13

13

13

9 Click13 Save13

10 To13 view13 the13 visualisaon13 click13 the13 Dashboard13 page13

Page of 30 71 Page of 30 71

13

11 Click13 the13 Add13 Visualiza0on13 icon13

13

12 Select13 the13 visiualizaon13 that13 you13 just13 created13

13

Page of 31 71 Page of 31 71

13

13 To13 save13 the13 dashboard13 click13 the13 Save13 Dashboard13 icon13

Page of 32 71 Page of 32 71

13

14 Add13 a13 tle13 and13 select13 whether13 to13 store13 the13 selected13 me13 frame13 with13 the13 dashboard13

13

15 Click13 Save13

16 You13 can13 edit13 the13 shape13 and13 number13 of13 charts13 and13 save13 the13 dashboard13

17 To13 re-shy‐run13 the13 dashboard13 select13 the13 Dashboard13 page13 and13 click13 the13 Load13 Saved13 Dashboard13 icon13

Page of 33 71 Page of 33 71

13

18 Select13 the13 saved13 dashboard13 from13 the13 list13

13

19 The13 dashboard13 runs13

13

Page of 34 71 Page of 34 71

Page of 35 71 Page of 35 71

Aler0ng13 with13 ELK13 ELK13 examines13 the13 log13 records13 in13 process13 to13 send13 alerts13 to13 third-shy‐party13 alerng13 event13 management13 and13 social13 media13 applicaons13 As13 shown13 previously13 in13 the13 ELK13 architecture13 the13 ELK13 strategy13 for13 alerng13 is13 to13 send13 alerts13 from13 the13 Logstash13 component13 directly13 Therefore13 log13 records13 and13 alerts13 are13 sent13 to13 both13 Elascsearch13 and13 third-shy‐party13 tools13 at13 the13 same13 instant13 using13 the13 output13 secon13 of13 the13 Logstash13 configuraon13 Any13 alerts13 are13 generated13 during13 the13 filter13 secon13 processing13 of13 the13 Logstash13 configuraon13

Create13 alerts13 in13 Logstash13 The13 Logstash13 filter13 secon13 is13 used13 to13 trigger13 alerts13 based13 on13 the13 log13 records13 being13 processed13 For13 example13 log13 records13 can13 be13 checked13 for13 errors13 warnings13 crashed13 or13 stopped13 messages13 and13 generate13 an13 alert13

Here13 is13 an13 example13 Logstash13 filter13 code13

13 if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

The13 specific13 tags13 are13 set13 against13 the13 log13 records13 to13 trigger13 the13 alert13 later13 in13 the13 configuraon13 in13 the13 output13 secon13

Control13 the13 number13 of13 generated13 alerts13 using13 the13 Logstash13 throcle13 plugin13 so13 that13 either13 a13 lot13 of13 instances13 in13 a13 fixed13 me13 period13 are13 required13 before13 an13 alert13 is13 sent13 or13 a13 specific13 number13 of13 instances13 triggers13 the13 alert13

Page of 36 71 Page of 36 71

Edit13 the13 plugin13 parameters13 to13 meet13 system13 requirements13

13 13 13 13 13 if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 10

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle13

after_count =gt 20

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 30

period =gt 1200

key =gt servicename_alarm_warning

Page of 37 71 Page of 37 71

add_tag =gt servicename_throttled

13 13 13 13 13 13 13 13 13

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 40

period =gt 1500

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

Enrich13 alerts13 with13 message13 detail13 Some13 of13 the13 third-shy‐party13 integraons13 benefit13 from13 having13 components13 of13 the13 raw13 log13 record13 sent13 in13 addion13 to13 the13 raw13 record13 or13 fields13 in13 the13 record13 edited13 transformed13 or13 translated13

Grok13 Logstash13 allows13 the13 log13 record13 to13 be13 broken13 up13 into13 logical13 secons13 and13 each13 secon13 stored13 in13 a13 new13 parameter13 using13 the13 grok13 parameter13 in13 the13 filter13 secon13 Here13 is13 an13 example13 grok13

13 grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

To13 build13 the13 grok13 statement13 Logstash13 offers13 an13 online13 tool13 at13 hcpsgrokdebugherokuappcom13

Translate13 If13 specific13 fields13 in13 the13 log13 record13 are13 coded13 (for13 example13 the13 Bluemix13 app13 name13 is13 oaen13 not13 sent13 in13 the13 syslog13 messages13 instead13 the13 GUID13 is13 sent13 which13 is13 not13 useful)13 then13 the13 filter13 secon13 translate13 plugin13 can13 be13 used13 to13 add13 new13 fields13 by13 holding13 the13 decoded13 versions13 of13 those13 values13

13 translate

field =gt data1

destination =gt data1

override =gt true

Page of 38 71 Page of 38 71

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

dc50f6e4-c052-436a-bf27-95ba392c4228 monverifyservicemanagement

0c677e82-5bfc-4e38-9155-e416249e3ee6 ms-catalog-postpyretic-pomiculture

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

However13 the13 translate13 plugin13 is13 not13 included13 with13 the13 Logstash13 installaon13 In13 the13 next13 secon13 I13 will13 13 describe13 how13 to13 add13 new13 plugins13 to13 the13 Logstash13 instance13

Add13 custom13 plugins13 to13 Logstash13 As13 an13 open13 source13 tool13 Logstash13 encourages13 users13 to13 develop13 content13 for13 all13 users13 to13 share13 Some13 of13 the13 plugins13 that13 were13 created13 by13 our13 community13 include13 a13 translaon13 plugin13 an13 output13 Slack13 plugin13 and13 an13 output13 syslog13 plugin13 Letrsquos13 look13 at13 how13 to13 add13 these13 plugins13 to13 your13 environment13

Filter13 translate13 plugin13 To13 add13 the13 translate13 plugin13

1 Download13 the13 plugin13

2 Copy13 the13 plugin13 file13 logstash-shy‐filter-shy‐translate-shy‐214gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-filter-translate 214

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Output13 Slack13 plugin13 To13 add13 the13 Slack13 plugin13

Page of 39 71 Page of 39 71

1 Download13 the13 plugin13 13

2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐slack-shy‐014gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-output-slack 014

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Output13 syslog13 plugin13 To13 add13 the13 syslog13 plugin13

1 Download13 the13 plugin13 13

2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐syslog-shy‐214gem13 to13 the13 ELK13 Virtual13 Server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-output-syslog 214

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Integrate13 Logstash13 with13 Slack13 To13 send13 generated13 alerts13 to13 Slack13 complete13 these13 steps13

1 Create13 a13 Slack13 integraon13 13

2 Use13 the13 Slack13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 Slack13 integraon13

113 Login13 to13 Slack13 and13 go13 to13 the13 Apps13 page13 13

Page of 40 71 Page of 40 71

13

313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13

13

A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13

13

Page of 42 71 Page of 42 71

The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13

13

The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13

13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

slack

url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5

channel =gt cloud-operations-mon

username =gt nick_cawood

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13

Page of 43 71 Page of 43 71

Page of 44 71 Page of 44 71

In13 Slack13 the13 alerts13 look13 like13 this13

13

Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13

1 Create13 a13 PagerDuty13 integraon13 13

2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13

1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13

13

2 Click13 Add13 New13 Service13

13

313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13

Page of 45 71 Page of 45 71

13

413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13

513 Click13 Add13 Service13

13

The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13

Page of 46 71 Page of 46 71

13

The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt be7eb1ea9c224018923e818d3e891d0b

incident_key =gt logstashhosttype

Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13

Page of 47 71 Page of 47 71

13

Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13

1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13

2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13

3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13

4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13

13 13 http

url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo

http_method =gt post

format =gt json

For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13

13 13 syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

Page of 48 71 Page of 48 71

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13

13

The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13

if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2

This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13

Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13

Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13

Logstash13 severity Shown13 in13 Omnibus13 as

alert Indeterminate

crical Warning

debug 7

emergency Clear

error Minor

informaonal 6

noce Crical

warning Major

Page of 49 71 Page of 49 71

13

Page of 50 71 Page of 50 71

Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13

input

tcp

port =gt 5000

type =gt syslog

filter

mutate

add_field =gt format =gt cf

grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

translate

field =gt data1

destination =gt data1

override =gt true

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

=== Check Alarm Criteria ===

Standard Key Words

if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

=== Alarm Exclusions ===

if [message] =~ (i)sometexthere

mutate remove_tag =gt servicename_alarm

Throttle alarms so we do not hit PD rate limits

Page of 52 71 Page of 52 71

if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 5

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle

after_count =gt 5

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 5

period =gt 1200

key =gt servicename_alarm_warning

add_tag =gt servicename_throttled

Page of 53 71 Page of 53 71

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

output

elasticsearch

hosts =gt localhost

stdout codec =gt rubydebug

if servicename_alarm_err in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

Page of 54 71 Page of 54 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 55 71 Page of 55 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

Page of 56 71 Page of 56 71

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert err message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_error in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

Page of 57 71 Page of 57 71

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 58 71 Page of 58 71

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 59 71 Page of 59 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert error message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

Page of 60 71 Page of 60 71

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

Page of 61 71 Page of 61 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 62 71 Page of 62 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt x

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert fatal message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

Page of 63 71 Page of 63 71

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

Page of 64 71 Page of 64 71

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 65 71 Page of 65 71

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt heavy_exclamation_mark

Page of 66 71 Page of 66 71

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert WARNING message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 67 71 Page of 67 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

Page of 68 71 Page of 68 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 69 71 Page of 69 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

Page of 70 71 Page of 70 71

Page of 71 71 Page of 71 71

Page 21: How to integrate the Elasticsearch Logstash Kibana (ELK ... · How to integrate the Elasticsearch Logstash Kibana (ELK) log analytics stack into IBM Bluemix Nick Cawood IBM Cloud

Configuraon13 is13 OK13

Start13 Logstash13 Aaer13 you13 have13 wricen13 and13 verified13 the13 Logstash13 configuraon13 you13 can13 start13 Logstash13 to13 process13 records13 from13 the13 Bluemix13 apps13 (via13 Cloud13 Foundry)13 Run13 the13 following13 as13 root13 user13

[rootelkforcase]$ cd homelogstashlogstash-233

[rootelkforcase]$ binlogstash agent -f configloggregatorconf

Logstash13 is13 now13 started13

Monitor13 Logstash13 processing13 Once13 Logstash13 is13 running13 and13 log13 records13 are13 being13 produced13 by13 the13 Bluemix13 apps13 you13 can13 monitor13 progress13 in13 the13 same13 window13 that13 was13 used13 to13 start13 Logstash13 Output13 is13 similar13 to13 the13 following13

13

The13 log13 records13 being13 broadcast13 from13 the13 Bluemix13 apps13 are13 now13 sent13 to13 Elascsearch13 from13 the13 Logstash13 output13 configuraon13 Records13 are13 sent13 in13 real13 me13 If13 Logstash13 is13 not13 running13 while13 log13 records13 are13 being13 produced13 then13 these13 records13 will13 not13 be13 processed13 by13 Logstash13 and13 will13 not13 be13 stored13 in13 Elascsearch13

Page of 21 71 Page of 21 71

Use13 ELK13 The13 log13 records13 from13 the13 Bluemix13 apps13 are13 now13 being13 stored13 and13 indexed13 by13 the13 Elascsearch13 component13 This13 data13 can13 be13 accessed13 via13 the13 Kibana13 applicaon13

Kibana13 Discover13 page13 Using13 a13 web13 browser13 either13 directly13 from13 a13 client13 or13 using13 the13 VNC13 client13 access13 Kibana13 using13 the13 publicIPaddress13 of13 the13 ELK13 virtual13 server13

hcpltpublicIPaddressgt560113

No13 userpassword13 credenals13 are13 required13

Kibana13 defaults13 to13 a13 search13 of13 all13 records13 ()13 in13 the13 last13 1513 minutes13 and13 returns13 the13 results13

13

More13 complex13 searches13 are13 possible13 by13 changing13 the13 me13 filter13 and13 supplying13 search13 queries13

Change13 the13 0me13 filter13 To13 change13 the13 me13 ranges13 click13 the13 clock13 symbol13 in13 the13 top-shy‐right13 of13 the13 Kibana13 Discover13 page13

13

You13 will13 see13 the13 following13 opons13

13

Page of 22 71 Page of 22 71

A13 variety13 of13 quick13 me13 ranges13 can13 be13 selected13 or13 you13 can13 switch13 to13 relave13 or13 absolute13 me13 ranges13

Quick13 me13 ranges13 allow13 switching13 between13 last13 X13 minutes13 to13 last13 Y13 hours13 or13 Z13 days13 etc13

Relave13 me13 ranges13 allow13 more13 specific13 versions13 of13 the13 Quick13 opons13 from13 specific13 minuteshoursdays13 to13 now13

13

Absolute13 me13 ranges13 specify13 the13 start13 and13 end13 mes13 for13 the13 searches13

13

Custom13 search13 queries13 You13 can13 use13 the13 text13 window13 of13 the13 Discover13 page13 to13 search13 for13 matches13 within13 the13 log13 records13 You13 can13 use13 Apache13 Lucene13 or13 Elascsearch13 query13 DSL13 syntax13

13

Page of 23 71 Page of 23 71

Saved13 searches13 You13 can13 save13 any13 custom13 search13 syntax13 and13 choice13 of13 me13 range13 for13 future13 ease13 of13 use13 Any13 relave13 or13 absolute13 me13 frame13 is13 saved13 as13 part13 of13 the13 search13 13

To13 save13 a13 search13

1 Set13 up13 and13 run13 the13 required13 syntax13 and13 me13 frame13

13

2 Click13 the13 Save13 Search13 icon13

13

3 Type13 in13 a13 name13 for13 the13 saved13 search13

Page of 24 71 Page of 24 71

13

7 To13 delete13 or13 update13 saved13 searches13 click13 13 manage13 saved13 searches13

13

Use13 the13 edit13 icons13 to13 make13 any13 changes13

Page of 26 71 Page of 26 71

13

Build13 simple13 Kibana13 histograms13 Kibana13 allows13 you13 to13 build13 charts13 and13 dashboards13 based13 on13 the13 search13 capability13 offered13 by13 the13 Discover13 page13 Saved13 searches13 can13 be13 used13 as13 the13 basis13 for13 charts13 and13 dashboards13 built13 using13 those13 charts13 To13 build13 a13 simple13 histogram13

1 Select13 the13 Visualize13 page13

13

2 Select13 Ver0cal13 Bar13 Chart13

13

3 To13 use13 a13 saved13 search13 click13 From13 a13 saved13 search13

Page of 27 71 Page of 27 71

13

4 Select13 a13 previously13 saved13 search13

13

5 Select13 to13 edit13 the13 x-shy‐axis

6 Edit13 the13 fields13 with13 informaon13 about13 the13 aggregaon13 field13 interval13 and13 custom13 label13

Page of 28 71 Page of 28 71

13

Page of 29 71 Page of 29 71

7 Click13 the13 green13 arrow13 The13 histogram13 is13 created13 and13 run13 based13 on13 the13 saved13 search

13

8 To13 save13 the13 histogram13 click13 the13 Save13 Visualiza0on13 icon13 and13 add13 a13 tle13

13

13

9 Click13 Save13

10 To13 view13 the13 visualisaon13 click13 the13 Dashboard13 page13

Page of 30 71 Page of 30 71

13

11 Click13 the13 Add13 Visualiza0on13 icon13

13

12 Select13 the13 visiualizaon13 that13 you13 just13 created13

13

Page of 31 71 Page of 31 71

13

13 To13 save13 the13 dashboard13 click13 the13 Save13 Dashboard13 icon13

Page of 32 71 Page of 32 71

13

14 Add13 a13 tle13 and13 select13 whether13 to13 store13 the13 selected13 me13 frame13 with13 the13 dashboard13

13

15 Click13 Save13

16 You13 can13 edit13 the13 shape13 and13 number13 of13 charts13 and13 save13 the13 dashboard13

17 To13 re-shy‐run13 the13 dashboard13 select13 the13 Dashboard13 page13 and13 click13 the13 Load13 Saved13 Dashboard13 icon13

Page of 33 71 Page of 33 71

13

18 Select13 the13 saved13 dashboard13 from13 the13 list13

13

19 The13 dashboard13 runs13

13

Page of 34 71 Page of 34 71

Page of 35 71 Page of 35 71

Aler0ng13 with13 ELK13 ELK13 examines13 the13 log13 records13 in13 process13 to13 send13 alerts13 to13 third-shy‐party13 alerng13 event13 management13 and13 social13 media13 applicaons13 As13 shown13 previously13 in13 the13 ELK13 architecture13 the13 ELK13 strategy13 for13 alerng13 is13 to13 send13 alerts13 from13 the13 Logstash13 component13 directly13 Therefore13 log13 records13 and13 alerts13 are13 sent13 to13 both13 Elascsearch13 and13 third-shy‐party13 tools13 at13 the13 same13 instant13 using13 the13 output13 secon13 of13 the13 Logstash13 configuraon13 Any13 alerts13 are13 generated13 during13 the13 filter13 secon13 processing13 of13 the13 Logstash13 configuraon13

Create13 alerts13 in13 Logstash13 The13 Logstash13 filter13 secon13 is13 used13 to13 trigger13 alerts13 based13 on13 the13 log13 records13 being13 processed13 For13 example13 log13 records13 can13 be13 checked13 for13 errors13 warnings13 crashed13 or13 stopped13 messages13 and13 generate13 an13 alert13

Here13 is13 an13 example13 Logstash13 filter13 code13

13 if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

The13 specific13 tags13 are13 set13 against13 the13 log13 records13 to13 trigger13 the13 alert13 later13 in13 the13 configuraon13 in13 the13 output13 secon13

Control13 the13 number13 of13 generated13 alerts13 using13 the13 Logstash13 throcle13 plugin13 so13 that13 either13 a13 lot13 of13 instances13 in13 a13 fixed13 me13 period13 are13 required13 before13 an13 alert13 is13 sent13 or13 a13 specific13 number13 of13 instances13 triggers13 the13 alert13

Page of 36 71 Page of 36 71

Edit13 the13 plugin13 parameters13 to13 meet13 system13 requirements13

13 13 13 13 13 if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 10

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle13

after_count =gt 20

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 30

period =gt 1200

key =gt servicename_alarm_warning

Page of 37 71 Page of 37 71

add_tag =gt servicename_throttled

13 13 13 13 13 13 13 13 13

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 40

period =gt 1500

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

Enrich13 alerts13 with13 message13 detail13 Some13 of13 the13 third-shy‐party13 integraons13 benefit13 from13 having13 components13 of13 the13 raw13 log13 record13 sent13 in13 addion13 to13 the13 raw13 record13 or13 fields13 in13 the13 record13 edited13 transformed13 or13 translated13

Grok13 Logstash13 allows13 the13 log13 record13 to13 be13 broken13 up13 into13 logical13 secons13 and13 each13 secon13 stored13 in13 a13 new13 parameter13 using13 the13 grok13 parameter13 in13 the13 filter13 secon13 Here13 is13 an13 example13 grok13

13 grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

To13 build13 the13 grok13 statement13 Logstash13 offers13 an13 online13 tool13 at13 hcpsgrokdebugherokuappcom13

Translate13 If13 specific13 fields13 in13 the13 log13 record13 are13 coded13 (for13 example13 the13 Bluemix13 app13 name13 is13 oaen13 not13 sent13 in13 the13 syslog13 messages13 instead13 the13 GUID13 is13 sent13 which13 is13 not13 useful)13 then13 the13 filter13 secon13 translate13 plugin13 can13 be13 used13 to13 add13 new13 fields13 by13 holding13 the13 decoded13 versions13 of13 those13 values13

13 translate

field =gt data1

destination =gt data1

override =gt true

Page of 38 71 Page of 38 71

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

dc50f6e4-c052-436a-bf27-95ba392c4228 monverifyservicemanagement

0c677e82-5bfc-4e38-9155-e416249e3ee6 ms-catalog-postpyretic-pomiculture

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

However13 the13 translate13 plugin13 is13 not13 included13 with13 the13 Logstash13 installaon13 In13 the13 next13 secon13 I13 will13 13 describe13 how13 to13 add13 new13 plugins13 to13 the13 Logstash13 instance13

Add13 custom13 plugins13 to13 Logstash13 As13 an13 open13 source13 tool13 Logstash13 encourages13 users13 to13 develop13 content13 for13 all13 users13 to13 share13 Some13 of13 the13 plugins13 that13 were13 created13 by13 our13 community13 include13 a13 translaon13 plugin13 an13 output13 Slack13 plugin13 and13 an13 output13 syslog13 plugin13 Letrsquos13 look13 at13 how13 to13 add13 these13 plugins13 to13 your13 environment13

Filter13 translate13 plugin13 To13 add13 the13 translate13 plugin13

1 Download13 the13 plugin13

2 Copy13 the13 plugin13 file13 logstash-shy‐filter-shy‐translate-shy‐214gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-filter-translate 214

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Output13 Slack13 plugin13 To13 add13 the13 Slack13 plugin13

Page of 39 71 Page of 39 71

1 Download13 the13 plugin13 13

2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐slack-shy‐014gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-output-slack 014

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Output13 syslog13 plugin13 To13 add13 the13 syslog13 plugin13

1 Download13 the13 plugin13 13

2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐syslog-shy‐214gem13 to13 the13 ELK13 Virtual13 Server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-output-syslog 214

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Integrate13 Logstash13 with13 Slack13 To13 send13 generated13 alerts13 to13 Slack13 complete13 these13 steps13

1 Create13 a13 Slack13 integraon13 13

2 Use13 the13 Slack13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 Slack13 integraon13

113 Login13 to13 Slack13 and13 go13 to13 the13 Apps13 page13 13

Page of 40 71 Page of 40 71

13

313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13

13

A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13

13

Page of 42 71 Page of 42 71

The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13

13

The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13

13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

slack

url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5

channel =gt cloud-operations-mon

username =gt nick_cawood

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13

Page of 43 71 Page of 43 71

Page of 44 71 Page of 44 71

In13 Slack13 the13 alerts13 look13 like13 this13

13

Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13

1 Create13 a13 PagerDuty13 integraon13 13

2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13

1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13

13

2 Click13 Add13 New13 Service13

13

313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13

Page of 45 71 Page of 45 71

13

413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13

513 Click13 Add13 Service13

13

The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13

Page of 46 71 Page of 46 71

13

The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt be7eb1ea9c224018923e818d3e891d0b

incident_key =gt logstashhosttype

Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13

Page of 47 71 Page of 47 71

13

Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13

1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13

2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13

3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13

4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13

13 13 http

url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo

http_method =gt post

format =gt json

For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13

13 13 syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

Page of 48 71 Page of 48 71

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13

13

The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13

if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2

This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13

Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13

Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13

Logstash13 severity Shown13 in13 Omnibus13 as

alert Indeterminate

crical Warning

debug 7

emergency Clear

error Minor

informaonal 6

noce Crical

warning Major

Page of 49 71 Page of 49 71

13

Page of 50 71 Page of 50 71

Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13

input

tcp

port =gt 5000

type =gt syslog

filter

mutate

add_field =gt format =gt cf

grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

translate

field =gt data1

destination =gt data1

override =gt true

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

=== Check Alarm Criteria ===

Standard Key Words

if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

=== Alarm Exclusions ===

if [message] =~ (i)sometexthere

mutate remove_tag =gt servicename_alarm

Throttle alarms so we do not hit PD rate limits

Page of 52 71 Page of 52 71

if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 5

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle

after_count =gt 5

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 5

period =gt 1200

key =gt servicename_alarm_warning

add_tag =gt servicename_throttled

Page of 53 71 Page of 53 71

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

output

elasticsearch

hosts =gt localhost

stdout codec =gt rubydebug

if servicename_alarm_err in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

Page of 54 71 Page of 54 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 55 71 Page of 55 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

Page of 56 71 Page of 56 71

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert err message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_error in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

Page of 57 71 Page of 57 71

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 58 71 Page of 58 71

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 59 71 Page of 59 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert error message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

Page of 60 71 Page of 60 71

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

Page of 61 71 Page of 61 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 62 71 Page of 62 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt x

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert fatal message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

Page of 63 71 Page of 63 71

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

Page of 64 71 Page of 64 71

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 65 71 Page of 65 71

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt heavy_exclamation_mark

Page of 66 71 Page of 66 71

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert WARNING message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 67 71 Page of 67 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

Page of 68 71 Page of 68 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 69 71 Page of 69 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

Page of 70 71 Page of 70 71

Page of 71 71 Page of 71 71

Page 22: How to integrate the Elasticsearch Logstash Kibana (ELK ... · How to integrate the Elasticsearch Logstash Kibana (ELK) log analytics stack into IBM Bluemix Nick Cawood IBM Cloud

Use13 ELK13 The13 log13 records13 from13 the13 Bluemix13 apps13 are13 now13 being13 stored13 and13 indexed13 by13 the13 Elascsearch13 component13 This13 data13 can13 be13 accessed13 via13 the13 Kibana13 applicaon13

Kibana13 Discover13 page13 Using13 a13 web13 browser13 either13 directly13 from13 a13 client13 or13 using13 the13 VNC13 client13 access13 Kibana13 using13 the13 publicIPaddress13 of13 the13 ELK13 virtual13 server13

hcpltpublicIPaddressgt560113

No13 userpassword13 credenals13 are13 required13

Kibana13 defaults13 to13 a13 search13 of13 all13 records13 ()13 in13 the13 last13 1513 minutes13 and13 returns13 the13 results13

13

More13 complex13 searches13 are13 possible13 by13 changing13 the13 me13 filter13 and13 supplying13 search13 queries13

Change13 the13 0me13 filter13 To13 change13 the13 me13 ranges13 click13 the13 clock13 symbol13 in13 the13 top-shy‐right13 of13 the13 Kibana13 Discover13 page13

13

You13 will13 see13 the13 following13 opons13

13

Page of 22 71 Page of 22 71

A13 variety13 of13 quick13 me13 ranges13 can13 be13 selected13 or13 you13 can13 switch13 to13 relave13 or13 absolute13 me13 ranges13

Quick13 me13 ranges13 allow13 switching13 between13 last13 X13 minutes13 to13 last13 Y13 hours13 or13 Z13 days13 etc13

Relave13 me13 ranges13 allow13 more13 specific13 versions13 of13 the13 Quick13 opons13 from13 specific13 minuteshoursdays13 to13 now13

13

Absolute13 me13 ranges13 specify13 the13 start13 and13 end13 mes13 for13 the13 searches13

13

Custom13 search13 queries13 You13 can13 use13 the13 text13 window13 of13 the13 Discover13 page13 to13 search13 for13 matches13 within13 the13 log13 records13 You13 can13 use13 Apache13 Lucene13 or13 Elascsearch13 query13 DSL13 syntax13

13

Page of 23 71 Page of 23 71

Saved13 searches13 You13 can13 save13 any13 custom13 search13 syntax13 and13 choice13 of13 me13 range13 for13 future13 ease13 of13 use13 Any13 relave13 or13 absolute13 me13 frame13 is13 saved13 as13 part13 of13 the13 search13 13

To13 save13 a13 search13

1 Set13 up13 and13 run13 the13 required13 syntax13 and13 me13 frame13

13

2 Click13 the13 Save13 Search13 icon13

13

3 Type13 in13 a13 name13 for13 the13 saved13 search13

Page of 24 71 Page of 24 71

13

7 To13 delete13 or13 update13 saved13 searches13 click13 13 manage13 saved13 searches13

13

Use13 the13 edit13 icons13 to13 make13 any13 changes13

Page of 26 71 Page of 26 71

13

Build13 simple13 Kibana13 histograms13 Kibana13 allows13 you13 to13 build13 charts13 and13 dashboards13 based13 on13 the13 search13 capability13 offered13 by13 the13 Discover13 page13 Saved13 searches13 can13 be13 used13 as13 the13 basis13 for13 charts13 and13 dashboards13 built13 using13 those13 charts13 To13 build13 a13 simple13 histogram13

1 Select13 the13 Visualize13 page13

13

2 Select13 Ver0cal13 Bar13 Chart13

13

3 To13 use13 a13 saved13 search13 click13 From13 a13 saved13 search13

Page of 27 71 Page of 27 71

13

4 Select13 a13 previously13 saved13 search13

13

5 Select13 to13 edit13 the13 x-shy‐axis

6 Edit13 the13 fields13 with13 informaon13 about13 the13 aggregaon13 field13 interval13 and13 custom13 label13

Page of 28 71 Page of 28 71

13

Page of 29 71 Page of 29 71

7 Click13 the13 green13 arrow13 The13 histogram13 is13 created13 and13 run13 based13 on13 the13 saved13 search

13

8 To13 save13 the13 histogram13 click13 the13 Save13 Visualiza0on13 icon13 and13 add13 a13 tle13

13

13

9 Click13 Save13

10 To13 view13 the13 visualisaon13 click13 the13 Dashboard13 page13

Page of 30 71 Page of 30 71

13

11 Click13 the13 Add13 Visualiza0on13 icon13

13

12 Select13 the13 visiualizaon13 that13 you13 just13 created13

13

Page of 31 71 Page of 31 71

13

13 To13 save13 the13 dashboard13 click13 the13 Save13 Dashboard13 icon13

Page of 32 71 Page of 32 71

13

14 Add13 a13 tle13 and13 select13 whether13 to13 store13 the13 selected13 me13 frame13 with13 the13 dashboard13

13

15 Click13 Save13

16 You13 can13 edit13 the13 shape13 and13 number13 of13 charts13 and13 save13 the13 dashboard13

17 To13 re-shy‐run13 the13 dashboard13 select13 the13 Dashboard13 page13 and13 click13 the13 Load13 Saved13 Dashboard13 icon13

Page of 33 71 Page of 33 71

13

18 Select13 the13 saved13 dashboard13 from13 the13 list13

13

19 The13 dashboard13 runs13

13

Page of 34 71 Page of 34 71

Page of 35 71 Page of 35 71

Aler0ng13 with13 ELK13 ELK13 examines13 the13 log13 records13 in13 process13 to13 send13 alerts13 to13 third-shy‐party13 alerng13 event13 management13 and13 social13 media13 applicaons13 As13 shown13 previously13 in13 the13 ELK13 architecture13 the13 ELK13 strategy13 for13 alerng13 is13 to13 send13 alerts13 from13 the13 Logstash13 component13 directly13 Therefore13 log13 records13 and13 alerts13 are13 sent13 to13 both13 Elascsearch13 and13 third-shy‐party13 tools13 at13 the13 same13 instant13 using13 the13 output13 secon13 of13 the13 Logstash13 configuraon13 Any13 alerts13 are13 generated13 during13 the13 filter13 secon13 processing13 of13 the13 Logstash13 configuraon13

Create13 alerts13 in13 Logstash13 The13 Logstash13 filter13 secon13 is13 used13 to13 trigger13 alerts13 based13 on13 the13 log13 records13 being13 processed13 For13 example13 log13 records13 can13 be13 checked13 for13 errors13 warnings13 crashed13 or13 stopped13 messages13 and13 generate13 an13 alert13

Here13 is13 an13 example13 Logstash13 filter13 code13

13 if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

The13 specific13 tags13 are13 set13 against13 the13 log13 records13 to13 trigger13 the13 alert13 later13 in13 the13 configuraon13 in13 the13 output13 secon13

Control13 the13 number13 of13 generated13 alerts13 using13 the13 Logstash13 throcle13 plugin13 so13 that13 either13 a13 lot13 of13 instances13 in13 a13 fixed13 me13 period13 are13 required13 before13 an13 alert13 is13 sent13 or13 a13 specific13 number13 of13 instances13 triggers13 the13 alert13

Page of 36 71 Page of 36 71

Edit13 the13 plugin13 parameters13 to13 meet13 system13 requirements13

13 13 13 13 13 if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 10

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle13

after_count =gt 20

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 30

period =gt 1200

key =gt servicename_alarm_warning

Page of 37 71 Page of 37 71

add_tag =gt servicename_throttled

13 13 13 13 13 13 13 13 13

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 40

period =gt 1500

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

Enrich13 alerts13 with13 message13 detail13 Some13 of13 the13 third-shy‐party13 integraons13 benefit13 from13 having13 components13 of13 the13 raw13 log13 record13 sent13 in13 addion13 to13 the13 raw13 record13 or13 fields13 in13 the13 record13 edited13 transformed13 or13 translated13

Grok13 Logstash13 allows13 the13 log13 record13 to13 be13 broken13 up13 into13 logical13 secons13 and13 each13 secon13 stored13 in13 a13 new13 parameter13 using13 the13 grok13 parameter13 in13 the13 filter13 secon13 Here13 is13 an13 example13 grok13

13 grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

To13 build13 the13 grok13 statement13 Logstash13 offers13 an13 online13 tool13 at13 hcpsgrokdebugherokuappcom13

Translate13 If13 specific13 fields13 in13 the13 log13 record13 are13 coded13 (for13 example13 the13 Bluemix13 app13 name13 is13 oaen13 not13 sent13 in13 the13 syslog13 messages13 instead13 the13 GUID13 is13 sent13 which13 is13 not13 useful)13 then13 the13 filter13 secon13 translate13 plugin13 can13 be13 used13 to13 add13 new13 fields13 by13 holding13 the13 decoded13 versions13 of13 those13 values13

13 translate

field =gt data1

destination =gt data1

override =gt true

Page of 38 71 Page of 38 71

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

dc50f6e4-c052-436a-bf27-95ba392c4228 monverifyservicemanagement

0c677e82-5bfc-4e38-9155-e416249e3ee6 ms-catalog-postpyretic-pomiculture

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

However13 the13 translate13 plugin13 is13 not13 included13 with13 the13 Logstash13 installaon13 In13 the13 next13 secon13 I13 will13 13 describe13 how13 to13 add13 new13 plugins13 to13 the13 Logstash13 instance13

Add13 custom13 plugins13 to13 Logstash13 As13 an13 open13 source13 tool13 Logstash13 encourages13 users13 to13 develop13 content13 for13 all13 users13 to13 share13 Some13 of13 the13 plugins13 that13 were13 created13 by13 our13 community13 include13 a13 translaon13 plugin13 an13 output13 Slack13 plugin13 and13 an13 output13 syslog13 plugin13 Letrsquos13 look13 at13 how13 to13 add13 these13 plugins13 to13 your13 environment13

Filter13 translate13 plugin13 To13 add13 the13 translate13 plugin13

1 Download13 the13 plugin13

2 Copy13 the13 plugin13 file13 logstash-shy‐filter-shy‐translate-shy‐214gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-filter-translate 214

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Output13 Slack13 plugin13 To13 add13 the13 Slack13 plugin13

Page of 39 71 Page of 39 71

1 Download13 the13 plugin13 13

2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐slack-shy‐014gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-output-slack 014

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Output13 syslog13 plugin13 To13 add13 the13 syslog13 plugin13

1 Download13 the13 plugin13 13

2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐syslog-shy‐214gem13 to13 the13 ELK13 Virtual13 Server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-output-syslog 214

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Integrate13 Logstash13 with13 Slack13 To13 send13 generated13 alerts13 to13 Slack13 complete13 these13 steps13

1 Create13 a13 Slack13 integraon13 13

2 Use13 the13 Slack13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 Slack13 integraon13

113 Login13 to13 Slack13 and13 go13 to13 the13 Apps13 page13 13

Page of 40 71 Page of 40 71

13

313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13

13

A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13

13

Page of 42 71 Page of 42 71

The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13

13

The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13

13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

slack

url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5

channel =gt cloud-operations-mon

username =gt nick_cawood

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13

Page of 43 71 Page of 43 71

Page of 44 71 Page of 44 71

In13 Slack13 the13 alerts13 look13 like13 this13

13

Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13

1 Create13 a13 PagerDuty13 integraon13 13

2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13

1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13

13

2 Click13 Add13 New13 Service13

13

313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13

Page of 45 71 Page of 45 71

13

413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13

513 Click13 Add13 Service13

13

The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13

Page of 46 71 Page of 46 71

13

The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt be7eb1ea9c224018923e818d3e891d0b

incident_key =gt logstashhosttype

Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13

Page of 47 71 Page of 47 71

13

Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13

1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13

2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13

3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13

4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13

13 13 http

url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo

http_method =gt post

format =gt json

For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13

13 13 syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

Page of 48 71 Page of 48 71

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13

13

The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13

if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2

This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13

Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13

Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13

Logstash13 severity Shown13 in13 Omnibus13 as

alert Indeterminate

crical Warning

debug 7

emergency Clear

error Minor

informaonal 6

noce Crical

warning Major

Page of 49 71 Page of 49 71

13

Page of 50 71 Page of 50 71

Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13

input

tcp

port =gt 5000

type =gt syslog

filter

mutate

add_field =gt format =gt cf

grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

translate

field =gt data1

destination =gt data1

override =gt true

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

=== Check Alarm Criteria ===

Standard Key Words

if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

=== Alarm Exclusions ===

if [message] =~ (i)sometexthere

mutate remove_tag =gt servicename_alarm

Throttle alarms so we do not hit PD rate limits

Page of 52 71 Page of 52 71

if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 5

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle

after_count =gt 5

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 5

period =gt 1200

key =gt servicename_alarm_warning

add_tag =gt servicename_throttled

Page of 53 71 Page of 53 71

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

output

elasticsearch

hosts =gt localhost

stdout codec =gt rubydebug

if servicename_alarm_err in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

Page of 54 71 Page of 54 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 55 71 Page of 55 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

Page of 56 71 Page of 56 71

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert err message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_error in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

Page of 57 71 Page of 57 71

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 58 71 Page of 58 71

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 59 71 Page of 59 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert error message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

Page of 60 71 Page of 60 71

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

Page of 61 71 Page of 61 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 62 71 Page of 62 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt x

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert fatal message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

Page of 63 71 Page of 63 71

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

Page of 64 71 Page of 64 71

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 65 71 Page of 65 71

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt heavy_exclamation_mark

Page of 66 71 Page of 66 71

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert WARNING message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 67 71 Page of 67 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

Page of 68 71 Page of 68 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 69 71 Page of 69 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

Page of 70 71 Page of 70 71

Page of 71 71 Page of 71 71

Page 23: How to integrate the Elasticsearch Logstash Kibana (ELK ... · How to integrate the Elasticsearch Logstash Kibana (ELK) log analytics stack into IBM Bluemix Nick Cawood IBM Cloud

A13 variety13 of13 quick13 me13 ranges13 can13 be13 selected13 or13 you13 can13 switch13 to13 relave13 or13 absolute13 me13 ranges13

Quick13 me13 ranges13 allow13 switching13 between13 last13 X13 minutes13 to13 last13 Y13 hours13 or13 Z13 days13 etc13

Relave13 me13 ranges13 allow13 more13 specific13 versions13 of13 the13 Quick13 opons13 from13 specific13 minuteshoursdays13 to13 now13

13

Absolute13 me13 ranges13 specify13 the13 start13 and13 end13 mes13 for13 the13 searches13

13

Custom13 search13 queries13 You13 can13 use13 the13 text13 window13 of13 the13 Discover13 page13 to13 search13 for13 matches13 within13 the13 log13 records13 You13 can13 use13 Apache13 Lucene13 or13 Elascsearch13 query13 DSL13 syntax13

13

Page of 23 71 Page of 23 71

Saved13 searches13 You13 can13 save13 any13 custom13 search13 syntax13 and13 choice13 of13 me13 range13 for13 future13 ease13 of13 use13 Any13 relave13 or13 absolute13 me13 frame13 is13 saved13 as13 part13 of13 the13 search13 13

To13 save13 a13 search13

1 Set13 up13 and13 run13 the13 required13 syntax13 and13 me13 frame13

13

2 Click13 the13 Save13 Search13 icon13

13

3 Type13 in13 a13 name13 for13 the13 saved13 search13

Page of 24 71 Page of 24 71

13

7 To13 delete13 or13 update13 saved13 searches13 click13 13 manage13 saved13 searches13

13

Use13 the13 edit13 icons13 to13 make13 any13 changes13

Page of 26 71 Page of 26 71

13

Build13 simple13 Kibana13 histograms13 Kibana13 allows13 you13 to13 build13 charts13 and13 dashboards13 based13 on13 the13 search13 capability13 offered13 by13 the13 Discover13 page13 Saved13 searches13 can13 be13 used13 as13 the13 basis13 for13 charts13 and13 dashboards13 built13 using13 those13 charts13 To13 build13 a13 simple13 histogram13

1 Select13 the13 Visualize13 page13

13

2 Select13 Ver0cal13 Bar13 Chart13

13

3 To13 use13 a13 saved13 search13 click13 From13 a13 saved13 search13

Page of 27 71 Page of 27 71

13

4 Select13 a13 previously13 saved13 search13

13

5 Select13 to13 edit13 the13 x-shy‐axis

6 Edit13 the13 fields13 with13 informaon13 about13 the13 aggregaon13 field13 interval13 and13 custom13 label13

Page of 28 71 Page of 28 71

13

Page of 29 71 Page of 29 71

7 Click13 the13 green13 arrow13 The13 histogram13 is13 created13 and13 run13 based13 on13 the13 saved13 search

13

8 To13 save13 the13 histogram13 click13 the13 Save13 Visualiza0on13 icon13 and13 add13 a13 tle13

13

13

9 Click13 Save13

10 To13 view13 the13 visualisaon13 click13 the13 Dashboard13 page13

Page of 30 71 Page of 30 71

13

11 Click13 the13 Add13 Visualiza0on13 icon13

13

12 Select13 the13 visiualizaon13 that13 you13 just13 created13

13

Page of 31 71 Page of 31 71

13

13 To13 save13 the13 dashboard13 click13 the13 Save13 Dashboard13 icon13

Page of 32 71 Page of 32 71

13

14 Add13 a13 tle13 and13 select13 whether13 to13 store13 the13 selected13 me13 frame13 with13 the13 dashboard13

13

15 Click13 Save13

16 You13 can13 edit13 the13 shape13 and13 number13 of13 charts13 and13 save13 the13 dashboard13

17 To13 re-shy‐run13 the13 dashboard13 select13 the13 Dashboard13 page13 and13 click13 the13 Load13 Saved13 Dashboard13 icon13

Page of 33 71 Page of 33 71

13

18 Select13 the13 saved13 dashboard13 from13 the13 list13

13

19 The13 dashboard13 runs13

13

Page of 34 71 Page of 34 71

Page of 35 71 Page of 35 71

Aler0ng13 with13 ELK13 ELK13 examines13 the13 log13 records13 in13 process13 to13 send13 alerts13 to13 third-shy‐party13 alerng13 event13 management13 and13 social13 media13 applicaons13 As13 shown13 previously13 in13 the13 ELK13 architecture13 the13 ELK13 strategy13 for13 alerng13 is13 to13 send13 alerts13 from13 the13 Logstash13 component13 directly13 Therefore13 log13 records13 and13 alerts13 are13 sent13 to13 both13 Elascsearch13 and13 third-shy‐party13 tools13 at13 the13 same13 instant13 using13 the13 output13 secon13 of13 the13 Logstash13 configuraon13 Any13 alerts13 are13 generated13 during13 the13 filter13 secon13 processing13 of13 the13 Logstash13 configuraon13

Create13 alerts13 in13 Logstash13 The13 Logstash13 filter13 secon13 is13 used13 to13 trigger13 alerts13 based13 on13 the13 log13 records13 being13 processed13 For13 example13 log13 records13 can13 be13 checked13 for13 errors13 warnings13 crashed13 or13 stopped13 messages13 and13 generate13 an13 alert13

Here13 is13 an13 example13 Logstash13 filter13 code13

13 if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

The13 specific13 tags13 are13 set13 against13 the13 log13 records13 to13 trigger13 the13 alert13 later13 in13 the13 configuraon13 in13 the13 output13 secon13

Control13 the13 number13 of13 generated13 alerts13 using13 the13 Logstash13 throcle13 plugin13 so13 that13 either13 a13 lot13 of13 instances13 in13 a13 fixed13 me13 period13 are13 required13 before13 an13 alert13 is13 sent13 or13 a13 specific13 number13 of13 instances13 triggers13 the13 alert13

Page of 36 71 Page of 36 71

Edit13 the13 plugin13 parameters13 to13 meet13 system13 requirements13

13 13 13 13 13 if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 10

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle13

after_count =gt 20

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 30

period =gt 1200

key =gt servicename_alarm_warning

Page of 37 71 Page of 37 71

add_tag =gt servicename_throttled

13 13 13 13 13 13 13 13 13

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 40

period =gt 1500

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

Enrich13 alerts13 with13 message13 detail13 Some13 of13 the13 third-shy‐party13 integraons13 benefit13 from13 having13 components13 of13 the13 raw13 log13 record13 sent13 in13 addion13 to13 the13 raw13 record13 or13 fields13 in13 the13 record13 edited13 transformed13 or13 translated13

Grok13 Logstash13 allows13 the13 log13 record13 to13 be13 broken13 up13 into13 logical13 secons13 and13 each13 secon13 stored13 in13 a13 new13 parameter13 using13 the13 grok13 parameter13 in13 the13 filter13 secon13 Here13 is13 an13 example13 grok13

13 grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

To13 build13 the13 grok13 statement13 Logstash13 offers13 an13 online13 tool13 at13 hcpsgrokdebugherokuappcom13

Translate13 If13 specific13 fields13 in13 the13 log13 record13 are13 coded13 (for13 example13 the13 Bluemix13 app13 name13 is13 oaen13 not13 sent13 in13 the13 syslog13 messages13 instead13 the13 GUID13 is13 sent13 which13 is13 not13 useful)13 then13 the13 filter13 secon13 translate13 plugin13 can13 be13 used13 to13 add13 new13 fields13 by13 holding13 the13 decoded13 versions13 of13 those13 values13

13 translate

field =gt data1

destination =gt data1

override =gt true

Page of 38 71 Page of 38 71

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

dc50f6e4-c052-436a-bf27-95ba392c4228 monverifyservicemanagement

0c677e82-5bfc-4e38-9155-e416249e3ee6 ms-catalog-postpyretic-pomiculture

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

However13 the13 translate13 plugin13 is13 not13 included13 with13 the13 Logstash13 installaon13 In13 the13 next13 secon13 I13 will13 13 describe13 how13 to13 add13 new13 plugins13 to13 the13 Logstash13 instance13

Add13 custom13 plugins13 to13 Logstash13 As13 an13 open13 source13 tool13 Logstash13 encourages13 users13 to13 develop13 content13 for13 all13 users13 to13 share13 Some13 of13 the13 plugins13 that13 were13 created13 by13 our13 community13 include13 a13 translaon13 plugin13 an13 output13 Slack13 plugin13 and13 an13 output13 syslog13 plugin13 Letrsquos13 look13 at13 how13 to13 add13 these13 plugins13 to13 your13 environment13

Filter13 translate13 plugin13 To13 add13 the13 translate13 plugin13

1 Download13 the13 plugin13

2 Copy13 the13 plugin13 file13 logstash-shy‐filter-shy‐translate-shy‐214gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-filter-translate 214

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Output13 Slack13 plugin13 To13 add13 the13 Slack13 plugin13

Page of 39 71 Page of 39 71

1 Download13 the13 plugin13 13

2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐slack-shy‐014gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-output-slack 014

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Output13 syslog13 plugin13 To13 add13 the13 syslog13 plugin13

1 Download13 the13 plugin13 13

2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐syslog-shy‐214gem13 to13 the13 ELK13 Virtual13 Server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-output-syslog 214

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Integrate13 Logstash13 with13 Slack13 To13 send13 generated13 alerts13 to13 Slack13 complete13 these13 steps13

1 Create13 a13 Slack13 integraon13 13

2 Use13 the13 Slack13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 Slack13 integraon13

113 Login13 to13 Slack13 and13 go13 to13 the13 Apps13 page13 13

Page of 40 71 Page of 40 71

13

313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13

13

A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13

13

Page of 42 71 Page of 42 71

The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13

13

The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13

13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

slack

url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5

channel =gt cloud-operations-mon

username =gt nick_cawood

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13

Page of 43 71 Page of 43 71

Page of 44 71 Page of 44 71

In13 Slack13 the13 alerts13 look13 like13 this13

13

Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13

1 Create13 a13 PagerDuty13 integraon13 13

2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13

1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13

13

2 Click13 Add13 New13 Service13

13

313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13

Page of 45 71 Page of 45 71

13

413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13

513 Click13 Add13 Service13

13

The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13

Page of 46 71 Page of 46 71

13

The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt be7eb1ea9c224018923e818d3e891d0b

incident_key =gt logstashhosttype

Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13

Page of 47 71 Page of 47 71

13

Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13

1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13

2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13

3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13

4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13

13 13 http

url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo

http_method =gt post

format =gt json

For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13

13 13 syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

Page of 48 71 Page of 48 71

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13

13

The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13

if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2

This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13

Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13

Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13

Logstash13 severity Shown13 in13 Omnibus13 as

alert Indeterminate

crical Warning

debug 7

emergency Clear

error Minor

informaonal 6

noce Crical

warning Major

Page of 49 71 Page of 49 71

13

Page of 50 71 Page of 50 71

Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13

input

tcp

port =gt 5000

type =gt syslog

filter

mutate

add_field =gt format =gt cf

grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

translate

field =gt data1

destination =gt data1

override =gt true

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

=== Check Alarm Criteria ===

Standard Key Words

if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

=== Alarm Exclusions ===

if [message] =~ (i)sometexthere

mutate remove_tag =gt servicename_alarm

Throttle alarms so we do not hit PD rate limits

Page of 52 71 Page of 52 71

if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 5

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle

after_count =gt 5

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 5

period =gt 1200

key =gt servicename_alarm_warning

add_tag =gt servicename_throttled

Page of 53 71 Page of 53 71

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

output

elasticsearch

hosts =gt localhost

stdout codec =gt rubydebug

if servicename_alarm_err in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

Page of 54 71 Page of 54 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 55 71 Page of 55 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

Page of 56 71 Page of 56 71

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert err message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_error in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

Page of 57 71 Page of 57 71

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 58 71 Page of 58 71

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 59 71 Page of 59 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert error message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

Page of 60 71 Page of 60 71

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

Page of 61 71 Page of 61 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 62 71 Page of 62 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt x

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert fatal message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

Page of 63 71 Page of 63 71

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

Page of 64 71 Page of 64 71

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 65 71 Page of 65 71

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt heavy_exclamation_mark

Page of 66 71 Page of 66 71

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert WARNING message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 67 71 Page of 67 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

Page of 68 71 Page of 68 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 69 71 Page of 69 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

Page of 70 71 Page of 70 71

Page of 71 71 Page of 71 71

Page 24: How to integrate the Elasticsearch Logstash Kibana (ELK ... · How to integrate the Elasticsearch Logstash Kibana (ELK) log analytics stack into IBM Bluemix Nick Cawood IBM Cloud

Saved13 searches13 You13 can13 save13 any13 custom13 search13 syntax13 and13 choice13 of13 me13 range13 for13 future13 ease13 of13 use13 Any13 relave13 or13 absolute13 me13 frame13 is13 saved13 as13 part13 of13 the13 search13 13

To13 save13 a13 search13

1 Set13 up13 and13 run13 the13 required13 syntax13 and13 me13 frame13

13

2 Click13 the13 Save13 Search13 icon13

13

3 Type13 in13 a13 name13 for13 the13 saved13 search13

Page of 24 71 Page of 24 71

13

7 To13 delete13 or13 update13 saved13 searches13 click13 13 manage13 saved13 searches13

13

Use13 the13 edit13 icons13 to13 make13 any13 changes13

Page of 26 71 Page of 26 71

13

Build13 simple13 Kibana13 histograms13 Kibana13 allows13 you13 to13 build13 charts13 and13 dashboards13 based13 on13 the13 search13 capability13 offered13 by13 the13 Discover13 page13 Saved13 searches13 can13 be13 used13 as13 the13 basis13 for13 charts13 and13 dashboards13 built13 using13 those13 charts13 To13 build13 a13 simple13 histogram13

1 Select13 the13 Visualize13 page13

13

2 Select13 Ver0cal13 Bar13 Chart13

13

3 To13 use13 a13 saved13 search13 click13 From13 a13 saved13 search13

Page of 27 71 Page of 27 71

13

4 Select13 a13 previously13 saved13 search13

13

5 Select13 to13 edit13 the13 x-shy‐axis

6 Edit13 the13 fields13 with13 informaon13 about13 the13 aggregaon13 field13 interval13 and13 custom13 label13

Page of 28 71 Page of 28 71

13

Page of 29 71 Page of 29 71

7 Click13 the13 green13 arrow13 The13 histogram13 is13 created13 and13 run13 based13 on13 the13 saved13 search

13

8 To13 save13 the13 histogram13 click13 the13 Save13 Visualiza0on13 icon13 and13 add13 a13 tle13

13

13

9 Click13 Save13

10 To13 view13 the13 visualisaon13 click13 the13 Dashboard13 page13

Page of 30 71 Page of 30 71

13

11 Click13 the13 Add13 Visualiza0on13 icon13

13

12 Select13 the13 visiualizaon13 that13 you13 just13 created13

13

Page of 31 71 Page of 31 71

13

13 To13 save13 the13 dashboard13 click13 the13 Save13 Dashboard13 icon13

Page of 32 71 Page of 32 71

13

14 Add13 a13 tle13 and13 select13 whether13 to13 store13 the13 selected13 me13 frame13 with13 the13 dashboard13

13

15 Click13 Save13

16 You13 can13 edit13 the13 shape13 and13 number13 of13 charts13 and13 save13 the13 dashboard13

17 To13 re-shy‐run13 the13 dashboard13 select13 the13 Dashboard13 page13 and13 click13 the13 Load13 Saved13 Dashboard13 icon13

Page of 33 71 Page of 33 71

13

18 Select13 the13 saved13 dashboard13 from13 the13 list13

13

19 The13 dashboard13 runs13

13

Page of 34 71 Page of 34 71

Page of 35 71 Page of 35 71

Aler0ng13 with13 ELK13 ELK13 examines13 the13 log13 records13 in13 process13 to13 send13 alerts13 to13 third-shy‐party13 alerng13 event13 management13 and13 social13 media13 applicaons13 As13 shown13 previously13 in13 the13 ELK13 architecture13 the13 ELK13 strategy13 for13 alerng13 is13 to13 send13 alerts13 from13 the13 Logstash13 component13 directly13 Therefore13 log13 records13 and13 alerts13 are13 sent13 to13 both13 Elascsearch13 and13 third-shy‐party13 tools13 at13 the13 same13 instant13 using13 the13 output13 secon13 of13 the13 Logstash13 configuraon13 Any13 alerts13 are13 generated13 during13 the13 filter13 secon13 processing13 of13 the13 Logstash13 configuraon13

Create13 alerts13 in13 Logstash13 The13 Logstash13 filter13 secon13 is13 used13 to13 trigger13 alerts13 based13 on13 the13 log13 records13 being13 processed13 For13 example13 log13 records13 can13 be13 checked13 for13 errors13 warnings13 crashed13 or13 stopped13 messages13 and13 generate13 an13 alert13

Here13 is13 an13 example13 Logstash13 filter13 code13

13 if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

The13 specific13 tags13 are13 set13 against13 the13 log13 records13 to13 trigger13 the13 alert13 later13 in13 the13 configuraon13 in13 the13 output13 secon13

Control13 the13 number13 of13 generated13 alerts13 using13 the13 Logstash13 throcle13 plugin13 so13 that13 either13 a13 lot13 of13 instances13 in13 a13 fixed13 me13 period13 are13 required13 before13 an13 alert13 is13 sent13 or13 a13 specific13 number13 of13 instances13 triggers13 the13 alert13

Page of 36 71 Page of 36 71

Edit13 the13 plugin13 parameters13 to13 meet13 system13 requirements13

13 13 13 13 13 if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 10

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle13

after_count =gt 20

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 30

period =gt 1200

key =gt servicename_alarm_warning

Page of 37 71 Page of 37 71

add_tag =gt servicename_throttled

13 13 13 13 13 13 13 13 13

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 40

period =gt 1500

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

Enrich13 alerts13 with13 message13 detail13 Some13 of13 the13 third-shy‐party13 integraons13 benefit13 from13 having13 components13 of13 the13 raw13 log13 record13 sent13 in13 addion13 to13 the13 raw13 record13 or13 fields13 in13 the13 record13 edited13 transformed13 or13 translated13

Grok13 Logstash13 allows13 the13 log13 record13 to13 be13 broken13 up13 into13 logical13 secons13 and13 each13 secon13 stored13 in13 a13 new13 parameter13 using13 the13 grok13 parameter13 in13 the13 filter13 secon13 Here13 is13 an13 example13 grok13

13 grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

To13 build13 the13 grok13 statement13 Logstash13 offers13 an13 online13 tool13 at13 hcpsgrokdebugherokuappcom13

Translate13 If13 specific13 fields13 in13 the13 log13 record13 are13 coded13 (for13 example13 the13 Bluemix13 app13 name13 is13 oaen13 not13 sent13 in13 the13 syslog13 messages13 instead13 the13 GUID13 is13 sent13 which13 is13 not13 useful)13 then13 the13 filter13 secon13 translate13 plugin13 can13 be13 used13 to13 add13 new13 fields13 by13 holding13 the13 decoded13 versions13 of13 those13 values13

13 translate

field =gt data1

destination =gt data1

override =gt true

Page of 38 71 Page of 38 71

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

dc50f6e4-c052-436a-bf27-95ba392c4228 monverifyservicemanagement

0c677e82-5bfc-4e38-9155-e416249e3ee6 ms-catalog-postpyretic-pomiculture

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

However13 the13 translate13 plugin13 is13 not13 included13 with13 the13 Logstash13 installaon13 In13 the13 next13 secon13 I13 will13 13 describe13 how13 to13 add13 new13 plugins13 to13 the13 Logstash13 instance13

Add13 custom13 plugins13 to13 Logstash13 As13 an13 open13 source13 tool13 Logstash13 encourages13 users13 to13 develop13 content13 for13 all13 users13 to13 share13 Some13 of13 the13 plugins13 that13 were13 created13 by13 our13 community13 include13 a13 translaon13 plugin13 an13 output13 Slack13 plugin13 and13 an13 output13 syslog13 plugin13 Letrsquos13 look13 at13 how13 to13 add13 these13 plugins13 to13 your13 environment13

Filter13 translate13 plugin13 To13 add13 the13 translate13 plugin13

1 Download13 the13 plugin13

2 Copy13 the13 plugin13 file13 logstash-shy‐filter-shy‐translate-shy‐214gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-filter-translate 214

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Output13 Slack13 plugin13 To13 add13 the13 Slack13 plugin13

Page of 39 71 Page of 39 71

1 Download13 the13 plugin13 13

2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐slack-shy‐014gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-output-slack 014

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Output13 syslog13 plugin13 To13 add13 the13 syslog13 plugin13

1 Download13 the13 plugin13 13

2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐syslog-shy‐214gem13 to13 the13 ELK13 Virtual13 Server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-output-syslog 214

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Integrate13 Logstash13 with13 Slack13 To13 send13 generated13 alerts13 to13 Slack13 complete13 these13 steps13

1 Create13 a13 Slack13 integraon13 13

2 Use13 the13 Slack13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 Slack13 integraon13

113 Login13 to13 Slack13 and13 go13 to13 the13 Apps13 page13 13

Page of 40 71 Page of 40 71

13

313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13

13

A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13

13

Page of 42 71 Page of 42 71

The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13

13

The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13

13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

slack

url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5

channel =gt cloud-operations-mon

username =gt nick_cawood

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13

Page of 43 71 Page of 43 71

Page of 44 71 Page of 44 71

In13 Slack13 the13 alerts13 look13 like13 this13

13

Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13

1 Create13 a13 PagerDuty13 integraon13 13

2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13

1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13

13

2 Click13 Add13 New13 Service13

13

313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13

Page of 45 71 Page of 45 71

13

413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13

513 Click13 Add13 Service13

13

The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13

Page of 46 71 Page of 46 71

13

The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt be7eb1ea9c224018923e818d3e891d0b

incident_key =gt logstashhosttype

Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13

Page of 47 71 Page of 47 71

13

Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13

1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13

2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13

3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13

4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13

13 13 http

url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo

http_method =gt post

format =gt json

For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13

13 13 syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

Page of 48 71 Page of 48 71

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13

13

The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13

if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2

This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13

Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13

Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13

Logstash13 severity Shown13 in13 Omnibus13 as

alert Indeterminate

crical Warning

debug 7

emergency Clear

error Minor

informaonal 6

noce Crical

warning Major

Page of 49 71 Page of 49 71

13

Page of 50 71 Page of 50 71

Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13

input

tcp

port =gt 5000

type =gt syslog

filter

mutate

add_field =gt format =gt cf

grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

translate

field =gt data1

destination =gt data1

override =gt true

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

=== Check Alarm Criteria ===

Standard Key Words

if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

=== Alarm Exclusions ===

if [message] =~ (i)sometexthere

mutate remove_tag =gt servicename_alarm

Throttle alarms so we do not hit PD rate limits

Page of 52 71 Page of 52 71

if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 5

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle

after_count =gt 5

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 5

period =gt 1200

key =gt servicename_alarm_warning

add_tag =gt servicename_throttled

Page of 53 71 Page of 53 71

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

output

elasticsearch

hosts =gt localhost

stdout codec =gt rubydebug

if servicename_alarm_err in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

Page of 54 71 Page of 54 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 55 71 Page of 55 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

Page of 56 71 Page of 56 71

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert err message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_error in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

Page of 57 71 Page of 57 71

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 58 71 Page of 58 71

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 59 71 Page of 59 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert error message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

Page of 60 71 Page of 60 71

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

Page of 61 71 Page of 61 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 62 71 Page of 62 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt x

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert fatal message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

Page of 63 71 Page of 63 71

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

Page of 64 71 Page of 64 71

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 65 71 Page of 65 71

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt heavy_exclamation_mark

Page of 66 71 Page of 66 71

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert WARNING message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 67 71 Page of 67 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

Page of 68 71 Page of 68 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 69 71 Page of 69 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

Page of 70 71 Page of 70 71

Page of 71 71 Page of 71 71

Page 25: How to integrate the Elasticsearch Logstash Kibana (ELK ... · How to integrate the Elasticsearch Logstash Kibana (ELK) log analytics stack into IBM Bluemix Nick Cawood IBM Cloud

13

7 To13 delete13 or13 update13 saved13 searches13 click13 13 manage13 saved13 searches13

13

Use13 the13 edit13 icons13 to13 make13 any13 changes13

Page of 26 71 Page of 26 71

13

Build13 simple13 Kibana13 histograms13 Kibana13 allows13 you13 to13 build13 charts13 and13 dashboards13 based13 on13 the13 search13 capability13 offered13 by13 the13 Discover13 page13 Saved13 searches13 can13 be13 used13 as13 the13 basis13 for13 charts13 and13 dashboards13 built13 using13 those13 charts13 To13 build13 a13 simple13 histogram13

1 Select13 the13 Visualize13 page13

13

2 Select13 Ver0cal13 Bar13 Chart13

13

3 To13 use13 a13 saved13 search13 click13 From13 a13 saved13 search13

Page of 27 71 Page of 27 71

13

4 Select13 a13 previously13 saved13 search13

13

5 Select13 to13 edit13 the13 x-shy‐axis

6 Edit13 the13 fields13 with13 informaon13 about13 the13 aggregaon13 field13 interval13 and13 custom13 label13

Page of 28 71 Page of 28 71

13

Page of 29 71 Page of 29 71

7 Click13 the13 green13 arrow13 The13 histogram13 is13 created13 and13 run13 based13 on13 the13 saved13 search

13

8 To13 save13 the13 histogram13 click13 the13 Save13 Visualiza0on13 icon13 and13 add13 a13 tle13

13

13

9 Click13 Save13

10 To13 view13 the13 visualisaon13 click13 the13 Dashboard13 page13

Page of 30 71 Page of 30 71

13

11 Click13 the13 Add13 Visualiza0on13 icon13

13

12 Select13 the13 visiualizaon13 that13 you13 just13 created13

13

Page of 31 71 Page of 31 71

13

13 To13 save13 the13 dashboard13 click13 the13 Save13 Dashboard13 icon13

Page of 32 71 Page of 32 71

13

14 Add13 a13 tle13 and13 select13 whether13 to13 store13 the13 selected13 me13 frame13 with13 the13 dashboard13

13

15 Click13 Save13

16 You13 can13 edit13 the13 shape13 and13 number13 of13 charts13 and13 save13 the13 dashboard13

17 To13 re-shy‐run13 the13 dashboard13 select13 the13 Dashboard13 page13 and13 click13 the13 Load13 Saved13 Dashboard13 icon13

Page of 33 71 Page of 33 71

13

18 Select13 the13 saved13 dashboard13 from13 the13 list13

13

19 The13 dashboard13 runs13

13

Page of 34 71 Page of 34 71

Page of 35 71 Page of 35 71

Aler0ng13 with13 ELK13 ELK13 examines13 the13 log13 records13 in13 process13 to13 send13 alerts13 to13 third-shy‐party13 alerng13 event13 management13 and13 social13 media13 applicaons13 As13 shown13 previously13 in13 the13 ELK13 architecture13 the13 ELK13 strategy13 for13 alerng13 is13 to13 send13 alerts13 from13 the13 Logstash13 component13 directly13 Therefore13 log13 records13 and13 alerts13 are13 sent13 to13 both13 Elascsearch13 and13 third-shy‐party13 tools13 at13 the13 same13 instant13 using13 the13 output13 secon13 of13 the13 Logstash13 configuraon13 Any13 alerts13 are13 generated13 during13 the13 filter13 secon13 processing13 of13 the13 Logstash13 configuraon13

Create13 alerts13 in13 Logstash13 The13 Logstash13 filter13 secon13 is13 used13 to13 trigger13 alerts13 based13 on13 the13 log13 records13 being13 processed13 For13 example13 log13 records13 can13 be13 checked13 for13 errors13 warnings13 crashed13 or13 stopped13 messages13 and13 generate13 an13 alert13

Here13 is13 an13 example13 Logstash13 filter13 code13

13 if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

The13 specific13 tags13 are13 set13 against13 the13 log13 records13 to13 trigger13 the13 alert13 later13 in13 the13 configuraon13 in13 the13 output13 secon13

Control13 the13 number13 of13 generated13 alerts13 using13 the13 Logstash13 throcle13 plugin13 so13 that13 either13 a13 lot13 of13 instances13 in13 a13 fixed13 me13 period13 are13 required13 before13 an13 alert13 is13 sent13 or13 a13 specific13 number13 of13 instances13 triggers13 the13 alert13

Page of 36 71 Page of 36 71

Edit13 the13 plugin13 parameters13 to13 meet13 system13 requirements13

13 13 13 13 13 if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 10

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle13

after_count =gt 20

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 30

period =gt 1200

key =gt servicename_alarm_warning

Page of 37 71 Page of 37 71

add_tag =gt servicename_throttled

13 13 13 13 13 13 13 13 13

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 40

period =gt 1500

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

Enrich13 alerts13 with13 message13 detail13 Some13 of13 the13 third-shy‐party13 integraons13 benefit13 from13 having13 components13 of13 the13 raw13 log13 record13 sent13 in13 addion13 to13 the13 raw13 record13 or13 fields13 in13 the13 record13 edited13 transformed13 or13 translated13

Grok13 Logstash13 allows13 the13 log13 record13 to13 be13 broken13 up13 into13 logical13 secons13 and13 each13 secon13 stored13 in13 a13 new13 parameter13 using13 the13 grok13 parameter13 in13 the13 filter13 secon13 Here13 is13 an13 example13 grok13

13 grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

To13 build13 the13 grok13 statement13 Logstash13 offers13 an13 online13 tool13 at13 hcpsgrokdebugherokuappcom13

Translate13 If13 specific13 fields13 in13 the13 log13 record13 are13 coded13 (for13 example13 the13 Bluemix13 app13 name13 is13 oaen13 not13 sent13 in13 the13 syslog13 messages13 instead13 the13 GUID13 is13 sent13 which13 is13 not13 useful)13 then13 the13 filter13 secon13 translate13 plugin13 can13 be13 used13 to13 add13 new13 fields13 by13 holding13 the13 decoded13 versions13 of13 those13 values13

13 translate

field =gt data1

destination =gt data1

override =gt true

Page of 38 71 Page of 38 71

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

dc50f6e4-c052-436a-bf27-95ba392c4228 monverifyservicemanagement

0c677e82-5bfc-4e38-9155-e416249e3ee6 ms-catalog-postpyretic-pomiculture

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

However13 the13 translate13 plugin13 is13 not13 included13 with13 the13 Logstash13 installaon13 In13 the13 next13 secon13 I13 will13 13 describe13 how13 to13 add13 new13 plugins13 to13 the13 Logstash13 instance13

Add13 custom13 plugins13 to13 Logstash13 As13 an13 open13 source13 tool13 Logstash13 encourages13 users13 to13 develop13 content13 for13 all13 users13 to13 share13 Some13 of13 the13 plugins13 that13 were13 created13 by13 our13 community13 include13 a13 translaon13 plugin13 an13 output13 Slack13 plugin13 and13 an13 output13 syslog13 plugin13 Letrsquos13 look13 at13 how13 to13 add13 these13 plugins13 to13 your13 environment13

Filter13 translate13 plugin13 To13 add13 the13 translate13 plugin13

1 Download13 the13 plugin13

2 Copy13 the13 plugin13 file13 logstash-shy‐filter-shy‐translate-shy‐214gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-filter-translate 214

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Output13 Slack13 plugin13 To13 add13 the13 Slack13 plugin13

Page of 39 71 Page of 39 71

1 Download13 the13 plugin13 13

2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐slack-shy‐014gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-output-slack 014

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Output13 syslog13 plugin13 To13 add13 the13 syslog13 plugin13

1 Download13 the13 plugin13 13

2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐syslog-shy‐214gem13 to13 the13 ELK13 Virtual13 Server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-output-syslog 214

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Integrate13 Logstash13 with13 Slack13 To13 send13 generated13 alerts13 to13 Slack13 complete13 these13 steps13

1 Create13 a13 Slack13 integraon13 13

2 Use13 the13 Slack13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 Slack13 integraon13

113 Login13 to13 Slack13 and13 go13 to13 the13 Apps13 page13 13

Page of 40 71 Page of 40 71

13

313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13

13

A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13

13

Page of 42 71 Page of 42 71

The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13

13

The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13

13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

slack

url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5

channel =gt cloud-operations-mon

username =gt nick_cawood

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13

Page of 43 71 Page of 43 71

Page of 44 71 Page of 44 71

In13 Slack13 the13 alerts13 look13 like13 this13

13

Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13

1 Create13 a13 PagerDuty13 integraon13 13

2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13

1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13

13

2 Click13 Add13 New13 Service13

13

313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13

Page of 45 71 Page of 45 71

13

413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13

513 Click13 Add13 Service13

13

The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13

Page of 46 71 Page of 46 71

13

The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt be7eb1ea9c224018923e818d3e891d0b

incident_key =gt logstashhosttype

Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13

Page of 47 71 Page of 47 71

13

Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13

1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13

2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13

3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13

4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13

13 13 http

url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo

http_method =gt post

format =gt json

For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13

13 13 syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

Page of 48 71 Page of 48 71

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13

13

The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13

if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2

This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13

Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13

Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13

Logstash13 severity Shown13 in13 Omnibus13 as

alert Indeterminate

crical Warning

debug 7

emergency Clear

error Minor

informaonal 6

noce Crical

warning Major

Page of 49 71 Page of 49 71

13

Page of 50 71 Page of 50 71

Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13

input

tcp

port =gt 5000

type =gt syslog

filter

mutate

add_field =gt format =gt cf

grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

translate

field =gt data1

destination =gt data1

override =gt true

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

=== Check Alarm Criteria ===

Standard Key Words

if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

=== Alarm Exclusions ===

if [message] =~ (i)sometexthere

mutate remove_tag =gt servicename_alarm

Throttle alarms so we do not hit PD rate limits

Page of 52 71 Page of 52 71

if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 5

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle

after_count =gt 5

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 5

period =gt 1200

key =gt servicename_alarm_warning

add_tag =gt servicename_throttled

Page of 53 71 Page of 53 71

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

output

elasticsearch

hosts =gt localhost

stdout codec =gt rubydebug

if servicename_alarm_err in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

Page of 54 71 Page of 54 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 55 71 Page of 55 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

Page of 56 71 Page of 56 71

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert err message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_error in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

Page of 57 71 Page of 57 71

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 58 71 Page of 58 71

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 59 71 Page of 59 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert error message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

Page of 60 71 Page of 60 71

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

Page of 61 71 Page of 61 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 62 71 Page of 62 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt x

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert fatal message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

Page of 63 71 Page of 63 71

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

Page of 64 71 Page of 64 71

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 65 71 Page of 65 71

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt heavy_exclamation_mark

Page of 66 71 Page of 66 71

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert WARNING message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 67 71 Page of 67 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

Page of 68 71 Page of 68 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 69 71 Page of 69 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

Page of 70 71 Page of 70 71

Page of 71 71 Page of 71 71

Page 26: How to integrate the Elasticsearch Logstash Kibana (ELK ... · How to integrate the Elasticsearch Logstash Kibana (ELK) log analytics stack into IBM Bluemix Nick Cawood IBM Cloud

13

Build13 simple13 Kibana13 histograms13 Kibana13 allows13 you13 to13 build13 charts13 and13 dashboards13 based13 on13 the13 search13 capability13 offered13 by13 the13 Discover13 page13 Saved13 searches13 can13 be13 used13 as13 the13 basis13 for13 charts13 and13 dashboards13 built13 using13 those13 charts13 To13 build13 a13 simple13 histogram13

1 Select13 the13 Visualize13 page13

13

2 Select13 Ver0cal13 Bar13 Chart13

13

3 To13 use13 a13 saved13 search13 click13 From13 a13 saved13 search13

Page of 27 71 Page of 27 71

13

4 Select13 a13 previously13 saved13 search13

13

5 Select13 to13 edit13 the13 x-shy‐axis

6 Edit13 the13 fields13 with13 informaon13 about13 the13 aggregaon13 field13 interval13 and13 custom13 label13

Page of 28 71 Page of 28 71

13

Page of 29 71 Page of 29 71

7 Click13 the13 green13 arrow13 The13 histogram13 is13 created13 and13 run13 based13 on13 the13 saved13 search

13

8 To13 save13 the13 histogram13 click13 the13 Save13 Visualiza0on13 icon13 and13 add13 a13 tle13

13

13

9 Click13 Save13

10 To13 view13 the13 visualisaon13 click13 the13 Dashboard13 page13

Page of 30 71 Page of 30 71

13

11 Click13 the13 Add13 Visualiza0on13 icon13

13

12 Select13 the13 visiualizaon13 that13 you13 just13 created13

13

Page of 31 71 Page of 31 71

13

13 To13 save13 the13 dashboard13 click13 the13 Save13 Dashboard13 icon13

Page of 32 71 Page of 32 71

13

14 Add13 a13 tle13 and13 select13 whether13 to13 store13 the13 selected13 me13 frame13 with13 the13 dashboard13

13

15 Click13 Save13

16 You13 can13 edit13 the13 shape13 and13 number13 of13 charts13 and13 save13 the13 dashboard13

17 To13 re-shy‐run13 the13 dashboard13 select13 the13 Dashboard13 page13 and13 click13 the13 Load13 Saved13 Dashboard13 icon13

Page of 33 71 Page of 33 71

13

18 Select13 the13 saved13 dashboard13 from13 the13 list13

13

19 The13 dashboard13 runs13

13

Page of 34 71 Page of 34 71

Page of 35 71 Page of 35 71

Aler0ng13 with13 ELK13 ELK13 examines13 the13 log13 records13 in13 process13 to13 send13 alerts13 to13 third-shy‐party13 alerng13 event13 management13 and13 social13 media13 applicaons13 As13 shown13 previously13 in13 the13 ELK13 architecture13 the13 ELK13 strategy13 for13 alerng13 is13 to13 send13 alerts13 from13 the13 Logstash13 component13 directly13 Therefore13 log13 records13 and13 alerts13 are13 sent13 to13 both13 Elascsearch13 and13 third-shy‐party13 tools13 at13 the13 same13 instant13 using13 the13 output13 secon13 of13 the13 Logstash13 configuraon13 Any13 alerts13 are13 generated13 during13 the13 filter13 secon13 processing13 of13 the13 Logstash13 configuraon13

Create13 alerts13 in13 Logstash13 The13 Logstash13 filter13 secon13 is13 used13 to13 trigger13 alerts13 based13 on13 the13 log13 records13 being13 processed13 For13 example13 log13 records13 can13 be13 checked13 for13 errors13 warnings13 crashed13 or13 stopped13 messages13 and13 generate13 an13 alert13

Here13 is13 an13 example13 Logstash13 filter13 code13

13 if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

The13 specific13 tags13 are13 set13 against13 the13 log13 records13 to13 trigger13 the13 alert13 later13 in13 the13 configuraon13 in13 the13 output13 secon13

Control13 the13 number13 of13 generated13 alerts13 using13 the13 Logstash13 throcle13 plugin13 so13 that13 either13 a13 lot13 of13 instances13 in13 a13 fixed13 me13 period13 are13 required13 before13 an13 alert13 is13 sent13 or13 a13 specific13 number13 of13 instances13 triggers13 the13 alert13

Page of 36 71 Page of 36 71

Edit13 the13 plugin13 parameters13 to13 meet13 system13 requirements13

13 13 13 13 13 if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 10

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle13

after_count =gt 20

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 30

period =gt 1200

key =gt servicename_alarm_warning

Page of 37 71 Page of 37 71

add_tag =gt servicename_throttled

13 13 13 13 13 13 13 13 13

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 40

period =gt 1500

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

Enrich13 alerts13 with13 message13 detail13 Some13 of13 the13 third-shy‐party13 integraons13 benefit13 from13 having13 components13 of13 the13 raw13 log13 record13 sent13 in13 addion13 to13 the13 raw13 record13 or13 fields13 in13 the13 record13 edited13 transformed13 or13 translated13

Grok13 Logstash13 allows13 the13 log13 record13 to13 be13 broken13 up13 into13 logical13 secons13 and13 each13 secon13 stored13 in13 a13 new13 parameter13 using13 the13 grok13 parameter13 in13 the13 filter13 secon13 Here13 is13 an13 example13 grok13

13 grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

To13 build13 the13 grok13 statement13 Logstash13 offers13 an13 online13 tool13 at13 hcpsgrokdebugherokuappcom13

Translate13 If13 specific13 fields13 in13 the13 log13 record13 are13 coded13 (for13 example13 the13 Bluemix13 app13 name13 is13 oaen13 not13 sent13 in13 the13 syslog13 messages13 instead13 the13 GUID13 is13 sent13 which13 is13 not13 useful)13 then13 the13 filter13 secon13 translate13 plugin13 can13 be13 used13 to13 add13 new13 fields13 by13 holding13 the13 decoded13 versions13 of13 those13 values13

13 translate

field =gt data1

destination =gt data1

override =gt true

Page of 38 71 Page of 38 71

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

dc50f6e4-c052-436a-bf27-95ba392c4228 monverifyservicemanagement

0c677e82-5bfc-4e38-9155-e416249e3ee6 ms-catalog-postpyretic-pomiculture

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

However13 the13 translate13 plugin13 is13 not13 included13 with13 the13 Logstash13 installaon13 In13 the13 next13 secon13 I13 will13 13 describe13 how13 to13 add13 new13 plugins13 to13 the13 Logstash13 instance13

Add13 custom13 plugins13 to13 Logstash13 As13 an13 open13 source13 tool13 Logstash13 encourages13 users13 to13 develop13 content13 for13 all13 users13 to13 share13 Some13 of13 the13 plugins13 that13 were13 created13 by13 our13 community13 include13 a13 translaon13 plugin13 an13 output13 Slack13 plugin13 and13 an13 output13 syslog13 plugin13 Letrsquos13 look13 at13 how13 to13 add13 these13 plugins13 to13 your13 environment13

Filter13 translate13 plugin13 To13 add13 the13 translate13 plugin13

1 Download13 the13 plugin13

2 Copy13 the13 plugin13 file13 logstash-shy‐filter-shy‐translate-shy‐214gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-filter-translate 214

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Output13 Slack13 plugin13 To13 add13 the13 Slack13 plugin13

Page of 39 71 Page of 39 71

1 Download13 the13 plugin13 13

2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐slack-shy‐014gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-output-slack 014

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Output13 syslog13 plugin13 To13 add13 the13 syslog13 plugin13

1 Download13 the13 plugin13 13

2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐syslog-shy‐214gem13 to13 the13 ELK13 Virtual13 Server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-output-syslog 214

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Integrate13 Logstash13 with13 Slack13 To13 send13 generated13 alerts13 to13 Slack13 complete13 these13 steps13

1 Create13 a13 Slack13 integraon13 13

2 Use13 the13 Slack13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 Slack13 integraon13

113 Login13 to13 Slack13 and13 go13 to13 the13 Apps13 page13 13

Page of 40 71 Page of 40 71

13

313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13

13

A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13

13

Page of 42 71 Page of 42 71

The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13

13

The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13

13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

slack

url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5

channel =gt cloud-operations-mon

username =gt nick_cawood

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13

Page of 43 71 Page of 43 71

Page of 44 71 Page of 44 71

In13 Slack13 the13 alerts13 look13 like13 this13

13

Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13

1 Create13 a13 PagerDuty13 integraon13 13

2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13

1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13

13

2 Click13 Add13 New13 Service13

13

313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13

Page of 45 71 Page of 45 71

13

413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13

513 Click13 Add13 Service13

13

The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13

Page of 46 71 Page of 46 71

13

The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt be7eb1ea9c224018923e818d3e891d0b

incident_key =gt logstashhosttype

Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13

Page of 47 71 Page of 47 71

13

Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13

1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13

2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13

3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13

4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13

13 13 http

url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo

http_method =gt post

format =gt json

For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13

13 13 syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

Page of 48 71 Page of 48 71

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13

13

The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13

if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2

This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13

Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13

Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13

Logstash13 severity Shown13 in13 Omnibus13 as

alert Indeterminate

crical Warning

debug 7

emergency Clear

error Minor

informaonal 6

noce Crical

warning Major

Page of 49 71 Page of 49 71

13

Page of 50 71 Page of 50 71

Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13

input

tcp

port =gt 5000

type =gt syslog

filter

mutate

add_field =gt format =gt cf

grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

translate

field =gt data1

destination =gt data1

override =gt true

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

=== Check Alarm Criteria ===

Standard Key Words

if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

=== Alarm Exclusions ===

if [message] =~ (i)sometexthere

mutate remove_tag =gt servicename_alarm

Throttle alarms so we do not hit PD rate limits

Page of 52 71 Page of 52 71

if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 5

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle

after_count =gt 5

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 5

period =gt 1200

key =gt servicename_alarm_warning

add_tag =gt servicename_throttled

Page of 53 71 Page of 53 71

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

output

elasticsearch

hosts =gt localhost

stdout codec =gt rubydebug

if servicename_alarm_err in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

Page of 54 71 Page of 54 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 55 71 Page of 55 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

Page of 56 71 Page of 56 71

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert err message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_error in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

Page of 57 71 Page of 57 71

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 58 71 Page of 58 71

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 59 71 Page of 59 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert error message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

Page of 60 71 Page of 60 71

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

Page of 61 71 Page of 61 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 62 71 Page of 62 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt x

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert fatal message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

Page of 63 71 Page of 63 71

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

Page of 64 71 Page of 64 71

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 65 71 Page of 65 71

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt heavy_exclamation_mark

Page of 66 71 Page of 66 71

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert WARNING message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 67 71 Page of 67 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

Page of 68 71 Page of 68 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 69 71 Page of 69 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

Page of 70 71 Page of 70 71

Page of 71 71 Page of 71 71

Page 27: How to integrate the Elasticsearch Logstash Kibana (ELK ... · How to integrate the Elasticsearch Logstash Kibana (ELK) log analytics stack into IBM Bluemix Nick Cawood IBM Cloud

13

4 Select13 a13 previously13 saved13 search13

13

5 Select13 to13 edit13 the13 x-shy‐axis

6 Edit13 the13 fields13 with13 informaon13 about13 the13 aggregaon13 field13 interval13 and13 custom13 label13

Page of 28 71 Page of 28 71

13

Page of 29 71 Page of 29 71

7 Click13 the13 green13 arrow13 The13 histogram13 is13 created13 and13 run13 based13 on13 the13 saved13 search

13

8 To13 save13 the13 histogram13 click13 the13 Save13 Visualiza0on13 icon13 and13 add13 a13 tle13

13

13

9 Click13 Save13

10 To13 view13 the13 visualisaon13 click13 the13 Dashboard13 page13

Page of 30 71 Page of 30 71

13

11 Click13 the13 Add13 Visualiza0on13 icon13

13

12 Select13 the13 visiualizaon13 that13 you13 just13 created13

13

Page of 31 71 Page of 31 71

13

13 To13 save13 the13 dashboard13 click13 the13 Save13 Dashboard13 icon13

Page of 32 71 Page of 32 71

13

14 Add13 a13 tle13 and13 select13 whether13 to13 store13 the13 selected13 me13 frame13 with13 the13 dashboard13

13

15 Click13 Save13

16 You13 can13 edit13 the13 shape13 and13 number13 of13 charts13 and13 save13 the13 dashboard13

17 To13 re-shy‐run13 the13 dashboard13 select13 the13 Dashboard13 page13 and13 click13 the13 Load13 Saved13 Dashboard13 icon13

Page of 33 71 Page of 33 71

13

18 Select13 the13 saved13 dashboard13 from13 the13 list13

13

19 The13 dashboard13 runs13

13

Page of 34 71 Page of 34 71

Page of 35 71 Page of 35 71

Aler0ng13 with13 ELK13 ELK13 examines13 the13 log13 records13 in13 process13 to13 send13 alerts13 to13 third-shy‐party13 alerng13 event13 management13 and13 social13 media13 applicaons13 As13 shown13 previously13 in13 the13 ELK13 architecture13 the13 ELK13 strategy13 for13 alerng13 is13 to13 send13 alerts13 from13 the13 Logstash13 component13 directly13 Therefore13 log13 records13 and13 alerts13 are13 sent13 to13 both13 Elascsearch13 and13 third-shy‐party13 tools13 at13 the13 same13 instant13 using13 the13 output13 secon13 of13 the13 Logstash13 configuraon13 Any13 alerts13 are13 generated13 during13 the13 filter13 secon13 processing13 of13 the13 Logstash13 configuraon13

Create13 alerts13 in13 Logstash13 The13 Logstash13 filter13 secon13 is13 used13 to13 trigger13 alerts13 based13 on13 the13 log13 records13 being13 processed13 For13 example13 log13 records13 can13 be13 checked13 for13 errors13 warnings13 crashed13 or13 stopped13 messages13 and13 generate13 an13 alert13

Here13 is13 an13 example13 Logstash13 filter13 code13

13 if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

The13 specific13 tags13 are13 set13 against13 the13 log13 records13 to13 trigger13 the13 alert13 later13 in13 the13 configuraon13 in13 the13 output13 secon13

Control13 the13 number13 of13 generated13 alerts13 using13 the13 Logstash13 throcle13 plugin13 so13 that13 either13 a13 lot13 of13 instances13 in13 a13 fixed13 me13 period13 are13 required13 before13 an13 alert13 is13 sent13 or13 a13 specific13 number13 of13 instances13 triggers13 the13 alert13

Page of 36 71 Page of 36 71

Edit13 the13 plugin13 parameters13 to13 meet13 system13 requirements13

13 13 13 13 13 if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 10

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle13

after_count =gt 20

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 30

period =gt 1200

key =gt servicename_alarm_warning

Page of 37 71 Page of 37 71

add_tag =gt servicename_throttled

13 13 13 13 13 13 13 13 13

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 40

period =gt 1500

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

Enrich13 alerts13 with13 message13 detail13 Some13 of13 the13 third-shy‐party13 integraons13 benefit13 from13 having13 components13 of13 the13 raw13 log13 record13 sent13 in13 addion13 to13 the13 raw13 record13 or13 fields13 in13 the13 record13 edited13 transformed13 or13 translated13

Grok13 Logstash13 allows13 the13 log13 record13 to13 be13 broken13 up13 into13 logical13 secons13 and13 each13 secon13 stored13 in13 a13 new13 parameter13 using13 the13 grok13 parameter13 in13 the13 filter13 secon13 Here13 is13 an13 example13 grok13

13 grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

To13 build13 the13 grok13 statement13 Logstash13 offers13 an13 online13 tool13 at13 hcpsgrokdebugherokuappcom13

Translate13 If13 specific13 fields13 in13 the13 log13 record13 are13 coded13 (for13 example13 the13 Bluemix13 app13 name13 is13 oaen13 not13 sent13 in13 the13 syslog13 messages13 instead13 the13 GUID13 is13 sent13 which13 is13 not13 useful)13 then13 the13 filter13 secon13 translate13 plugin13 can13 be13 used13 to13 add13 new13 fields13 by13 holding13 the13 decoded13 versions13 of13 those13 values13

13 translate

field =gt data1

destination =gt data1

override =gt true

Page of 38 71 Page of 38 71

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

dc50f6e4-c052-436a-bf27-95ba392c4228 monverifyservicemanagement

0c677e82-5bfc-4e38-9155-e416249e3ee6 ms-catalog-postpyretic-pomiculture

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

However13 the13 translate13 plugin13 is13 not13 included13 with13 the13 Logstash13 installaon13 In13 the13 next13 secon13 I13 will13 13 describe13 how13 to13 add13 new13 plugins13 to13 the13 Logstash13 instance13

Add13 custom13 plugins13 to13 Logstash13 As13 an13 open13 source13 tool13 Logstash13 encourages13 users13 to13 develop13 content13 for13 all13 users13 to13 share13 Some13 of13 the13 plugins13 that13 were13 created13 by13 our13 community13 include13 a13 translaon13 plugin13 an13 output13 Slack13 plugin13 and13 an13 output13 syslog13 plugin13 Letrsquos13 look13 at13 how13 to13 add13 these13 plugins13 to13 your13 environment13

Filter13 translate13 plugin13 To13 add13 the13 translate13 plugin13

1 Download13 the13 plugin13

2 Copy13 the13 plugin13 file13 logstash-shy‐filter-shy‐translate-shy‐214gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-filter-translate 214

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Output13 Slack13 plugin13 To13 add13 the13 Slack13 plugin13

Page of 39 71 Page of 39 71

1 Download13 the13 plugin13 13

2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐slack-shy‐014gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-output-slack 014

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Output13 syslog13 plugin13 To13 add13 the13 syslog13 plugin13

1 Download13 the13 plugin13 13

2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐syslog-shy‐214gem13 to13 the13 ELK13 Virtual13 Server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-output-syslog 214

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Integrate13 Logstash13 with13 Slack13 To13 send13 generated13 alerts13 to13 Slack13 complete13 these13 steps13

1 Create13 a13 Slack13 integraon13 13

2 Use13 the13 Slack13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 Slack13 integraon13

113 Login13 to13 Slack13 and13 go13 to13 the13 Apps13 page13 13

Page of 40 71 Page of 40 71

13

313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13

13

A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13

13

Page of 42 71 Page of 42 71

The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13

13

The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13

13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

slack

url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5

channel =gt cloud-operations-mon

username =gt nick_cawood

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13

Page of 43 71 Page of 43 71

Page of 44 71 Page of 44 71

In13 Slack13 the13 alerts13 look13 like13 this13

13

Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13

1 Create13 a13 PagerDuty13 integraon13 13

2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13

1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13

13

2 Click13 Add13 New13 Service13

13

313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13

Page of 45 71 Page of 45 71

13

413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13

513 Click13 Add13 Service13

13

The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13

Page of 46 71 Page of 46 71

13

The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt be7eb1ea9c224018923e818d3e891d0b

incident_key =gt logstashhosttype

Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13

Page of 47 71 Page of 47 71

13

Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13

1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13

2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13

3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13

4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13

13 13 http

url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo

http_method =gt post

format =gt json

For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13

13 13 syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

Page of 48 71 Page of 48 71

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13

13

The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13

if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2

This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13

Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13

Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13

Logstash13 severity Shown13 in13 Omnibus13 as

alert Indeterminate

crical Warning

debug 7

emergency Clear

error Minor

informaonal 6

noce Crical

warning Major

Page of 49 71 Page of 49 71

13

Page of 50 71 Page of 50 71

Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13

input

tcp

port =gt 5000

type =gt syslog

filter

mutate

add_field =gt format =gt cf

grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

translate

field =gt data1

destination =gt data1

override =gt true

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

=== Check Alarm Criteria ===

Standard Key Words

if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

=== Alarm Exclusions ===

if [message] =~ (i)sometexthere

mutate remove_tag =gt servicename_alarm

Throttle alarms so we do not hit PD rate limits

Page of 52 71 Page of 52 71

if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 5

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle

after_count =gt 5

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 5

period =gt 1200

key =gt servicename_alarm_warning

add_tag =gt servicename_throttled

Page of 53 71 Page of 53 71

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

output

elasticsearch

hosts =gt localhost

stdout codec =gt rubydebug

if servicename_alarm_err in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

Page of 54 71 Page of 54 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 55 71 Page of 55 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

Page of 56 71 Page of 56 71

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert err message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_error in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

Page of 57 71 Page of 57 71

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 58 71 Page of 58 71

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 59 71 Page of 59 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert error message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

Page of 60 71 Page of 60 71

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

Page of 61 71 Page of 61 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 62 71 Page of 62 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt x

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert fatal message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

Page of 63 71 Page of 63 71

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

Page of 64 71 Page of 64 71

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 65 71 Page of 65 71

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt heavy_exclamation_mark

Page of 66 71 Page of 66 71

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert WARNING message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 67 71 Page of 67 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

Page of 68 71 Page of 68 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 69 71 Page of 69 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

Page of 70 71 Page of 70 71

Page of 71 71 Page of 71 71

Page 28: How to integrate the Elasticsearch Logstash Kibana (ELK ... · How to integrate the Elasticsearch Logstash Kibana (ELK) log analytics stack into IBM Bluemix Nick Cawood IBM Cloud

13

Page of 29 71 Page of 29 71

7 Click13 the13 green13 arrow13 The13 histogram13 is13 created13 and13 run13 based13 on13 the13 saved13 search

13

8 To13 save13 the13 histogram13 click13 the13 Save13 Visualiza0on13 icon13 and13 add13 a13 tle13

13

13

9 Click13 Save13

10 To13 view13 the13 visualisaon13 click13 the13 Dashboard13 page13

Page of 30 71 Page of 30 71

13

11 Click13 the13 Add13 Visualiza0on13 icon13

13

12 Select13 the13 visiualizaon13 that13 you13 just13 created13

13

Page of 31 71 Page of 31 71

13

13 To13 save13 the13 dashboard13 click13 the13 Save13 Dashboard13 icon13

Page of 32 71 Page of 32 71

13

14 Add13 a13 tle13 and13 select13 whether13 to13 store13 the13 selected13 me13 frame13 with13 the13 dashboard13

13

15 Click13 Save13

16 You13 can13 edit13 the13 shape13 and13 number13 of13 charts13 and13 save13 the13 dashboard13

17 To13 re-shy‐run13 the13 dashboard13 select13 the13 Dashboard13 page13 and13 click13 the13 Load13 Saved13 Dashboard13 icon13

Page of 33 71 Page of 33 71

13

18 Select13 the13 saved13 dashboard13 from13 the13 list13

13

19 The13 dashboard13 runs13

13

Page of 34 71 Page of 34 71

Page of 35 71 Page of 35 71

Aler0ng13 with13 ELK13 ELK13 examines13 the13 log13 records13 in13 process13 to13 send13 alerts13 to13 third-shy‐party13 alerng13 event13 management13 and13 social13 media13 applicaons13 As13 shown13 previously13 in13 the13 ELK13 architecture13 the13 ELK13 strategy13 for13 alerng13 is13 to13 send13 alerts13 from13 the13 Logstash13 component13 directly13 Therefore13 log13 records13 and13 alerts13 are13 sent13 to13 both13 Elascsearch13 and13 third-shy‐party13 tools13 at13 the13 same13 instant13 using13 the13 output13 secon13 of13 the13 Logstash13 configuraon13 Any13 alerts13 are13 generated13 during13 the13 filter13 secon13 processing13 of13 the13 Logstash13 configuraon13

Create13 alerts13 in13 Logstash13 The13 Logstash13 filter13 secon13 is13 used13 to13 trigger13 alerts13 based13 on13 the13 log13 records13 being13 processed13 For13 example13 log13 records13 can13 be13 checked13 for13 errors13 warnings13 crashed13 or13 stopped13 messages13 and13 generate13 an13 alert13

Here13 is13 an13 example13 Logstash13 filter13 code13

13 if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

The13 specific13 tags13 are13 set13 against13 the13 log13 records13 to13 trigger13 the13 alert13 later13 in13 the13 configuraon13 in13 the13 output13 secon13

Control13 the13 number13 of13 generated13 alerts13 using13 the13 Logstash13 throcle13 plugin13 so13 that13 either13 a13 lot13 of13 instances13 in13 a13 fixed13 me13 period13 are13 required13 before13 an13 alert13 is13 sent13 or13 a13 specific13 number13 of13 instances13 triggers13 the13 alert13

Page of 36 71 Page of 36 71

Edit13 the13 plugin13 parameters13 to13 meet13 system13 requirements13

13 13 13 13 13 if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 10

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle13

after_count =gt 20

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 30

period =gt 1200

key =gt servicename_alarm_warning

Page of 37 71 Page of 37 71

add_tag =gt servicename_throttled

13 13 13 13 13 13 13 13 13

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 40

period =gt 1500

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

Enrich13 alerts13 with13 message13 detail13 Some13 of13 the13 third-shy‐party13 integraons13 benefit13 from13 having13 components13 of13 the13 raw13 log13 record13 sent13 in13 addion13 to13 the13 raw13 record13 or13 fields13 in13 the13 record13 edited13 transformed13 or13 translated13

Grok13 Logstash13 allows13 the13 log13 record13 to13 be13 broken13 up13 into13 logical13 secons13 and13 each13 secon13 stored13 in13 a13 new13 parameter13 using13 the13 grok13 parameter13 in13 the13 filter13 secon13 Here13 is13 an13 example13 grok13

13 grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

To13 build13 the13 grok13 statement13 Logstash13 offers13 an13 online13 tool13 at13 hcpsgrokdebugherokuappcom13

Translate13 If13 specific13 fields13 in13 the13 log13 record13 are13 coded13 (for13 example13 the13 Bluemix13 app13 name13 is13 oaen13 not13 sent13 in13 the13 syslog13 messages13 instead13 the13 GUID13 is13 sent13 which13 is13 not13 useful)13 then13 the13 filter13 secon13 translate13 plugin13 can13 be13 used13 to13 add13 new13 fields13 by13 holding13 the13 decoded13 versions13 of13 those13 values13

13 translate

field =gt data1

destination =gt data1

override =gt true

Page of 38 71 Page of 38 71

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

dc50f6e4-c052-436a-bf27-95ba392c4228 monverifyservicemanagement

0c677e82-5bfc-4e38-9155-e416249e3ee6 ms-catalog-postpyretic-pomiculture

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

However13 the13 translate13 plugin13 is13 not13 included13 with13 the13 Logstash13 installaon13 In13 the13 next13 secon13 I13 will13 13 describe13 how13 to13 add13 new13 plugins13 to13 the13 Logstash13 instance13

Add13 custom13 plugins13 to13 Logstash13 As13 an13 open13 source13 tool13 Logstash13 encourages13 users13 to13 develop13 content13 for13 all13 users13 to13 share13 Some13 of13 the13 plugins13 that13 were13 created13 by13 our13 community13 include13 a13 translaon13 plugin13 an13 output13 Slack13 plugin13 and13 an13 output13 syslog13 plugin13 Letrsquos13 look13 at13 how13 to13 add13 these13 plugins13 to13 your13 environment13

Filter13 translate13 plugin13 To13 add13 the13 translate13 plugin13

1 Download13 the13 plugin13

2 Copy13 the13 plugin13 file13 logstash-shy‐filter-shy‐translate-shy‐214gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-filter-translate 214

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Output13 Slack13 plugin13 To13 add13 the13 Slack13 plugin13

Page of 39 71 Page of 39 71

1 Download13 the13 plugin13 13

2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐slack-shy‐014gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-output-slack 014

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Output13 syslog13 plugin13 To13 add13 the13 syslog13 plugin13

1 Download13 the13 plugin13 13

2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐syslog-shy‐214gem13 to13 the13 ELK13 Virtual13 Server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-output-syslog 214

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Integrate13 Logstash13 with13 Slack13 To13 send13 generated13 alerts13 to13 Slack13 complete13 these13 steps13

1 Create13 a13 Slack13 integraon13 13

2 Use13 the13 Slack13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 Slack13 integraon13

113 Login13 to13 Slack13 and13 go13 to13 the13 Apps13 page13 13

Page of 40 71 Page of 40 71

13

313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13

13

A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13

13

Page of 42 71 Page of 42 71

The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13

13

The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13

13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

slack

url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5

channel =gt cloud-operations-mon

username =gt nick_cawood

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13

Page of 43 71 Page of 43 71

Page of 44 71 Page of 44 71

In13 Slack13 the13 alerts13 look13 like13 this13

13

Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13

1 Create13 a13 PagerDuty13 integraon13 13

2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13

1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13

13

2 Click13 Add13 New13 Service13

13

313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13

Page of 45 71 Page of 45 71

13

413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13

513 Click13 Add13 Service13

13

The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13

Page of 46 71 Page of 46 71

13

The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt be7eb1ea9c224018923e818d3e891d0b

incident_key =gt logstashhosttype

Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13

Page of 47 71 Page of 47 71

13

Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13

1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13

2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13

3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13

4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13

13 13 http

url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo

http_method =gt post

format =gt json

For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13

13 13 syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

Page of 48 71 Page of 48 71

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13

13

The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13

if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2

This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13

Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13

Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13

Logstash13 severity Shown13 in13 Omnibus13 as

alert Indeterminate

crical Warning

debug 7

emergency Clear

error Minor

informaonal 6

noce Crical

warning Major

Page of 49 71 Page of 49 71

13

Page of 50 71 Page of 50 71

Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13

input

tcp

port =gt 5000

type =gt syslog

filter

mutate

add_field =gt format =gt cf

grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

translate

field =gt data1

destination =gt data1

override =gt true

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

=== Check Alarm Criteria ===

Standard Key Words

if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

=== Alarm Exclusions ===

if [message] =~ (i)sometexthere

mutate remove_tag =gt servicename_alarm

Throttle alarms so we do not hit PD rate limits

Page of 52 71 Page of 52 71

if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 5

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle

after_count =gt 5

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 5

period =gt 1200

key =gt servicename_alarm_warning

add_tag =gt servicename_throttled

Page of 53 71 Page of 53 71

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

output

elasticsearch

hosts =gt localhost

stdout codec =gt rubydebug

if servicename_alarm_err in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

Page of 54 71 Page of 54 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 55 71 Page of 55 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

Page of 56 71 Page of 56 71

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert err message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_error in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

Page of 57 71 Page of 57 71

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 58 71 Page of 58 71

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 59 71 Page of 59 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert error message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

Page of 60 71 Page of 60 71

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

Page of 61 71 Page of 61 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 62 71 Page of 62 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt x

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert fatal message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

Page of 63 71 Page of 63 71

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

Page of 64 71 Page of 64 71

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 65 71 Page of 65 71

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt heavy_exclamation_mark

Page of 66 71 Page of 66 71

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert WARNING message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 67 71 Page of 67 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

Page of 68 71 Page of 68 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 69 71 Page of 69 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

Page of 70 71 Page of 70 71

Page of 71 71 Page of 71 71

Page 29: How to integrate the Elasticsearch Logstash Kibana (ELK ... · How to integrate the Elasticsearch Logstash Kibana (ELK) log analytics stack into IBM Bluemix Nick Cawood IBM Cloud

7 Click13 the13 green13 arrow13 The13 histogram13 is13 created13 and13 run13 based13 on13 the13 saved13 search

13

8 To13 save13 the13 histogram13 click13 the13 Save13 Visualiza0on13 icon13 and13 add13 a13 tle13

13

13

9 Click13 Save13

10 To13 view13 the13 visualisaon13 click13 the13 Dashboard13 page13

Page of 30 71 Page of 30 71

13

11 Click13 the13 Add13 Visualiza0on13 icon13

13

12 Select13 the13 visiualizaon13 that13 you13 just13 created13

13

Page of 31 71 Page of 31 71

13

13 To13 save13 the13 dashboard13 click13 the13 Save13 Dashboard13 icon13

Page of 32 71 Page of 32 71

13

14 Add13 a13 tle13 and13 select13 whether13 to13 store13 the13 selected13 me13 frame13 with13 the13 dashboard13

13

15 Click13 Save13

16 You13 can13 edit13 the13 shape13 and13 number13 of13 charts13 and13 save13 the13 dashboard13

17 To13 re-shy‐run13 the13 dashboard13 select13 the13 Dashboard13 page13 and13 click13 the13 Load13 Saved13 Dashboard13 icon13

Page of 33 71 Page of 33 71

13

18 Select13 the13 saved13 dashboard13 from13 the13 list13

13

19 The13 dashboard13 runs13

13

Page of 34 71 Page of 34 71

Page of 35 71 Page of 35 71

Aler0ng13 with13 ELK13 ELK13 examines13 the13 log13 records13 in13 process13 to13 send13 alerts13 to13 third-shy‐party13 alerng13 event13 management13 and13 social13 media13 applicaons13 As13 shown13 previously13 in13 the13 ELK13 architecture13 the13 ELK13 strategy13 for13 alerng13 is13 to13 send13 alerts13 from13 the13 Logstash13 component13 directly13 Therefore13 log13 records13 and13 alerts13 are13 sent13 to13 both13 Elascsearch13 and13 third-shy‐party13 tools13 at13 the13 same13 instant13 using13 the13 output13 secon13 of13 the13 Logstash13 configuraon13 Any13 alerts13 are13 generated13 during13 the13 filter13 secon13 processing13 of13 the13 Logstash13 configuraon13

Create13 alerts13 in13 Logstash13 The13 Logstash13 filter13 secon13 is13 used13 to13 trigger13 alerts13 based13 on13 the13 log13 records13 being13 processed13 For13 example13 log13 records13 can13 be13 checked13 for13 errors13 warnings13 crashed13 or13 stopped13 messages13 and13 generate13 an13 alert13

Here13 is13 an13 example13 Logstash13 filter13 code13

13 if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

The13 specific13 tags13 are13 set13 against13 the13 log13 records13 to13 trigger13 the13 alert13 later13 in13 the13 configuraon13 in13 the13 output13 secon13

Control13 the13 number13 of13 generated13 alerts13 using13 the13 Logstash13 throcle13 plugin13 so13 that13 either13 a13 lot13 of13 instances13 in13 a13 fixed13 me13 period13 are13 required13 before13 an13 alert13 is13 sent13 or13 a13 specific13 number13 of13 instances13 triggers13 the13 alert13

Page of 36 71 Page of 36 71

Edit13 the13 plugin13 parameters13 to13 meet13 system13 requirements13

13 13 13 13 13 if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 10

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle13

after_count =gt 20

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 30

period =gt 1200

key =gt servicename_alarm_warning

Page of 37 71 Page of 37 71

add_tag =gt servicename_throttled

13 13 13 13 13 13 13 13 13

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 40

period =gt 1500

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

Enrich13 alerts13 with13 message13 detail13 Some13 of13 the13 third-shy‐party13 integraons13 benefit13 from13 having13 components13 of13 the13 raw13 log13 record13 sent13 in13 addion13 to13 the13 raw13 record13 or13 fields13 in13 the13 record13 edited13 transformed13 or13 translated13

Grok13 Logstash13 allows13 the13 log13 record13 to13 be13 broken13 up13 into13 logical13 secons13 and13 each13 secon13 stored13 in13 a13 new13 parameter13 using13 the13 grok13 parameter13 in13 the13 filter13 secon13 Here13 is13 an13 example13 grok13

13 grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

To13 build13 the13 grok13 statement13 Logstash13 offers13 an13 online13 tool13 at13 hcpsgrokdebugherokuappcom13

Translate13 If13 specific13 fields13 in13 the13 log13 record13 are13 coded13 (for13 example13 the13 Bluemix13 app13 name13 is13 oaen13 not13 sent13 in13 the13 syslog13 messages13 instead13 the13 GUID13 is13 sent13 which13 is13 not13 useful)13 then13 the13 filter13 secon13 translate13 plugin13 can13 be13 used13 to13 add13 new13 fields13 by13 holding13 the13 decoded13 versions13 of13 those13 values13

13 translate

field =gt data1

destination =gt data1

override =gt true

Page of 38 71 Page of 38 71

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

dc50f6e4-c052-436a-bf27-95ba392c4228 monverifyservicemanagement

0c677e82-5bfc-4e38-9155-e416249e3ee6 ms-catalog-postpyretic-pomiculture

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

However13 the13 translate13 plugin13 is13 not13 included13 with13 the13 Logstash13 installaon13 In13 the13 next13 secon13 I13 will13 13 describe13 how13 to13 add13 new13 plugins13 to13 the13 Logstash13 instance13

Add13 custom13 plugins13 to13 Logstash13 As13 an13 open13 source13 tool13 Logstash13 encourages13 users13 to13 develop13 content13 for13 all13 users13 to13 share13 Some13 of13 the13 plugins13 that13 were13 created13 by13 our13 community13 include13 a13 translaon13 plugin13 an13 output13 Slack13 plugin13 and13 an13 output13 syslog13 plugin13 Letrsquos13 look13 at13 how13 to13 add13 these13 plugins13 to13 your13 environment13

Filter13 translate13 plugin13 To13 add13 the13 translate13 plugin13

1 Download13 the13 plugin13

2 Copy13 the13 plugin13 file13 logstash-shy‐filter-shy‐translate-shy‐214gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-filter-translate 214

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Output13 Slack13 plugin13 To13 add13 the13 Slack13 plugin13

Page of 39 71 Page of 39 71

1 Download13 the13 plugin13 13

2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐slack-shy‐014gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-output-slack 014

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Output13 syslog13 plugin13 To13 add13 the13 syslog13 plugin13

1 Download13 the13 plugin13 13

2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐syslog-shy‐214gem13 to13 the13 ELK13 Virtual13 Server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-output-syslog 214

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Integrate13 Logstash13 with13 Slack13 To13 send13 generated13 alerts13 to13 Slack13 complete13 these13 steps13

1 Create13 a13 Slack13 integraon13 13

2 Use13 the13 Slack13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 Slack13 integraon13

113 Login13 to13 Slack13 and13 go13 to13 the13 Apps13 page13 13

Page of 40 71 Page of 40 71

13

313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13

13

A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13

13

Page of 42 71 Page of 42 71

The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13

13

The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13

13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

slack

url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5

channel =gt cloud-operations-mon

username =gt nick_cawood

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13

Page of 43 71 Page of 43 71

Page of 44 71 Page of 44 71

In13 Slack13 the13 alerts13 look13 like13 this13

13

Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13

1 Create13 a13 PagerDuty13 integraon13 13

2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13

1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13

13

2 Click13 Add13 New13 Service13

13

313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13

Page of 45 71 Page of 45 71

13

413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13

513 Click13 Add13 Service13

13

The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13

Page of 46 71 Page of 46 71

13

The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt be7eb1ea9c224018923e818d3e891d0b

incident_key =gt logstashhosttype

Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13

Page of 47 71 Page of 47 71

13

Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13

1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13

2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13

3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13

4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13

13 13 http

url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo

http_method =gt post

format =gt json

For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13

13 13 syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

Page of 48 71 Page of 48 71

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13

13

The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13

if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2

This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13

Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13

Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13

Logstash13 severity Shown13 in13 Omnibus13 as

alert Indeterminate

crical Warning

debug 7

emergency Clear

error Minor

informaonal 6

noce Crical

warning Major

Page of 49 71 Page of 49 71

13

Page of 50 71 Page of 50 71

Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13

input

tcp

port =gt 5000

type =gt syslog

filter

mutate

add_field =gt format =gt cf

grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

translate

field =gt data1

destination =gt data1

override =gt true

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

=== Check Alarm Criteria ===

Standard Key Words

if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

=== Alarm Exclusions ===

if [message] =~ (i)sometexthere

mutate remove_tag =gt servicename_alarm

Throttle alarms so we do not hit PD rate limits

Page of 52 71 Page of 52 71

if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 5

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle

after_count =gt 5

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 5

period =gt 1200

key =gt servicename_alarm_warning

add_tag =gt servicename_throttled

Page of 53 71 Page of 53 71

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

output

elasticsearch

hosts =gt localhost

stdout codec =gt rubydebug

if servicename_alarm_err in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

Page of 54 71 Page of 54 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 55 71 Page of 55 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

Page of 56 71 Page of 56 71

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert err message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_error in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

Page of 57 71 Page of 57 71

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 58 71 Page of 58 71

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 59 71 Page of 59 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert error message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

Page of 60 71 Page of 60 71

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

Page of 61 71 Page of 61 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 62 71 Page of 62 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt x

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert fatal message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

Page of 63 71 Page of 63 71

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

Page of 64 71 Page of 64 71

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 65 71 Page of 65 71

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt heavy_exclamation_mark

Page of 66 71 Page of 66 71

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert WARNING message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 67 71 Page of 67 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

Page of 68 71 Page of 68 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 69 71 Page of 69 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

Page of 70 71 Page of 70 71

Page of 71 71 Page of 71 71

Page 30: How to integrate the Elasticsearch Logstash Kibana (ELK ... · How to integrate the Elasticsearch Logstash Kibana (ELK) log analytics stack into IBM Bluemix Nick Cawood IBM Cloud

13

11 Click13 the13 Add13 Visualiza0on13 icon13

13

12 Select13 the13 visiualizaon13 that13 you13 just13 created13

13

Page of 31 71 Page of 31 71

13

13 To13 save13 the13 dashboard13 click13 the13 Save13 Dashboard13 icon13

Page of 32 71 Page of 32 71

13

14 Add13 a13 tle13 and13 select13 whether13 to13 store13 the13 selected13 me13 frame13 with13 the13 dashboard13

13

15 Click13 Save13

16 You13 can13 edit13 the13 shape13 and13 number13 of13 charts13 and13 save13 the13 dashboard13

17 To13 re-shy‐run13 the13 dashboard13 select13 the13 Dashboard13 page13 and13 click13 the13 Load13 Saved13 Dashboard13 icon13

Page of 33 71 Page of 33 71

13

18 Select13 the13 saved13 dashboard13 from13 the13 list13

13

19 The13 dashboard13 runs13

13

Page of 34 71 Page of 34 71

Page of 35 71 Page of 35 71

Aler0ng13 with13 ELK13 ELK13 examines13 the13 log13 records13 in13 process13 to13 send13 alerts13 to13 third-shy‐party13 alerng13 event13 management13 and13 social13 media13 applicaons13 As13 shown13 previously13 in13 the13 ELK13 architecture13 the13 ELK13 strategy13 for13 alerng13 is13 to13 send13 alerts13 from13 the13 Logstash13 component13 directly13 Therefore13 log13 records13 and13 alerts13 are13 sent13 to13 both13 Elascsearch13 and13 third-shy‐party13 tools13 at13 the13 same13 instant13 using13 the13 output13 secon13 of13 the13 Logstash13 configuraon13 Any13 alerts13 are13 generated13 during13 the13 filter13 secon13 processing13 of13 the13 Logstash13 configuraon13

Create13 alerts13 in13 Logstash13 The13 Logstash13 filter13 secon13 is13 used13 to13 trigger13 alerts13 based13 on13 the13 log13 records13 being13 processed13 For13 example13 log13 records13 can13 be13 checked13 for13 errors13 warnings13 crashed13 or13 stopped13 messages13 and13 generate13 an13 alert13

Here13 is13 an13 example13 Logstash13 filter13 code13

13 if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

The13 specific13 tags13 are13 set13 against13 the13 log13 records13 to13 trigger13 the13 alert13 later13 in13 the13 configuraon13 in13 the13 output13 secon13

Control13 the13 number13 of13 generated13 alerts13 using13 the13 Logstash13 throcle13 plugin13 so13 that13 either13 a13 lot13 of13 instances13 in13 a13 fixed13 me13 period13 are13 required13 before13 an13 alert13 is13 sent13 or13 a13 specific13 number13 of13 instances13 triggers13 the13 alert13

Page of 36 71 Page of 36 71

Edit13 the13 plugin13 parameters13 to13 meet13 system13 requirements13

13 13 13 13 13 if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 10

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle13

after_count =gt 20

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 30

period =gt 1200

key =gt servicename_alarm_warning

Page of 37 71 Page of 37 71

add_tag =gt servicename_throttled

13 13 13 13 13 13 13 13 13

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 40

period =gt 1500

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

Enrich13 alerts13 with13 message13 detail13 Some13 of13 the13 third-shy‐party13 integraons13 benefit13 from13 having13 components13 of13 the13 raw13 log13 record13 sent13 in13 addion13 to13 the13 raw13 record13 or13 fields13 in13 the13 record13 edited13 transformed13 or13 translated13

Grok13 Logstash13 allows13 the13 log13 record13 to13 be13 broken13 up13 into13 logical13 secons13 and13 each13 secon13 stored13 in13 a13 new13 parameter13 using13 the13 grok13 parameter13 in13 the13 filter13 secon13 Here13 is13 an13 example13 grok13

13 grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

To13 build13 the13 grok13 statement13 Logstash13 offers13 an13 online13 tool13 at13 hcpsgrokdebugherokuappcom13

Translate13 If13 specific13 fields13 in13 the13 log13 record13 are13 coded13 (for13 example13 the13 Bluemix13 app13 name13 is13 oaen13 not13 sent13 in13 the13 syslog13 messages13 instead13 the13 GUID13 is13 sent13 which13 is13 not13 useful)13 then13 the13 filter13 secon13 translate13 plugin13 can13 be13 used13 to13 add13 new13 fields13 by13 holding13 the13 decoded13 versions13 of13 those13 values13

13 translate

field =gt data1

destination =gt data1

override =gt true

Page of 38 71 Page of 38 71

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

dc50f6e4-c052-436a-bf27-95ba392c4228 monverifyservicemanagement

0c677e82-5bfc-4e38-9155-e416249e3ee6 ms-catalog-postpyretic-pomiculture

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

However13 the13 translate13 plugin13 is13 not13 included13 with13 the13 Logstash13 installaon13 In13 the13 next13 secon13 I13 will13 13 describe13 how13 to13 add13 new13 plugins13 to13 the13 Logstash13 instance13

Add13 custom13 plugins13 to13 Logstash13 As13 an13 open13 source13 tool13 Logstash13 encourages13 users13 to13 develop13 content13 for13 all13 users13 to13 share13 Some13 of13 the13 plugins13 that13 were13 created13 by13 our13 community13 include13 a13 translaon13 plugin13 an13 output13 Slack13 plugin13 and13 an13 output13 syslog13 plugin13 Letrsquos13 look13 at13 how13 to13 add13 these13 plugins13 to13 your13 environment13

Filter13 translate13 plugin13 To13 add13 the13 translate13 plugin13

1 Download13 the13 plugin13

2 Copy13 the13 plugin13 file13 logstash-shy‐filter-shy‐translate-shy‐214gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-filter-translate 214

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Output13 Slack13 plugin13 To13 add13 the13 Slack13 plugin13

Page of 39 71 Page of 39 71

1 Download13 the13 plugin13 13

2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐slack-shy‐014gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-output-slack 014

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Output13 syslog13 plugin13 To13 add13 the13 syslog13 plugin13

1 Download13 the13 plugin13 13

2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐syslog-shy‐214gem13 to13 the13 ELK13 Virtual13 Server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-output-syslog 214

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Integrate13 Logstash13 with13 Slack13 To13 send13 generated13 alerts13 to13 Slack13 complete13 these13 steps13

1 Create13 a13 Slack13 integraon13 13

2 Use13 the13 Slack13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 Slack13 integraon13

113 Login13 to13 Slack13 and13 go13 to13 the13 Apps13 page13 13

Page of 40 71 Page of 40 71

13

313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13

13

A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13

13

Page of 42 71 Page of 42 71

The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13

13

The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13

13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

slack

url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5

channel =gt cloud-operations-mon

username =gt nick_cawood

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13

Page of 43 71 Page of 43 71

Page of 44 71 Page of 44 71

In13 Slack13 the13 alerts13 look13 like13 this13

13

Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13

1 Create13 a13 PagerDuty13 integraon13 13

2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13

1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13

13

2 Click13 Add13 New13 Service13

13

313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13

Page of 45 71 Page of 45 71

13

413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13

513 Click13 Add13 Service13

13

The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13

Page of 46 71 Page of 46 71

13

The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt be7eb1ea9c224018923e818d3e891d0b

incident_key =gt logstashhosttype

Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13

Page of 47 71 Page of 47 71

13

Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13

1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13

2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13

3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13

4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13

13 13 http

url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo

http_method =gt post

format =gt json

For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13

13 13 syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

Page of 48 71 Page of 48 71

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13

13

The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13

if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2

This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13

Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13

Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13

Logstash13 severity Shown13 in13 Omnibus13 as

alert Indeterminate

crical Warning

debug 7

emergency Clear

error Minor

informaonal 6

noce Crical

warning Major

Page of 49 71 Page of 49 71

13

Page of 50 71 Page of 50 71

Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13

input

tcp

port =gt 5000

type =gt syslog

filter

mutate

add_field =gt format =gt cf

grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

translate

field =gt data1

destination =gt data1

override =gt true

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

=== Check Alarm Criteria ===

Standard Key Words

if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

=== Alarm Exclusions ===

if [message] =~ (i)sometexthere

mutate remove_tag =gt servicename_alarm

Throttle alarms so we do not hit PD rate limits

Page of 52 71 Page of 52 71

if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 5

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle

after_count =gt 5

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 5

period =gt 1200

key =gt servicename_alarm_warning

add_tag =gt servicename_throttled

Page of 53 71 Page of 53 71

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

output

elasticsearch

hosts =gt localhost

stdout codec =gt rubydebug

if servicename_alarm_err in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

Page of 54 71 Page of 54 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 55 71 Page of 55 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

Page of 56 71 Page of 56 71

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert err message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_error in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

Page of 57 71 Page of 57 71

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 58 71 Page of 58 71

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 59 71 Page of 59 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert error message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

Page of 60 71 Page of 60 71

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

Page of 61 71 Page of 61 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 62 71 Page of 62 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt x

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert fatal message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

Page of 63 71 Page of 63 71

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

Page of 64 71 Page of 64 71

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 65 71 Page of 65 71

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt heavy_exclamation_mark

Page of 66 71 Page of 66 71

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert WARNING message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 67 71 Page of 67 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

Page of 68 71 Page of 68 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 69 71 Page of 69 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

Page of 70 71 Page of 70 71

Page of 71 71 Page of 71 71

Page 31: How to integrate the Elasticsearch Logstash Kibana (ELK ... · How to integrate the Elasticsearch Logstash Kibana (ELK) log analytics stack into IBM Bluemix Nick Cawood IBM Cloud

13

13 To13 save13 the13 dashboard13 click13 the13 Save13 Dashboard13 icon13

Page of 32 71 Page of 32 71

13

14 Add13 a13 tle13 and13 select13 whether13 to13 store13 the13 selected13 me13 frame13 with13 the13 dashboard13

13

15 Click13 Save13

16 You13 can13 edit13 the13 shape13 and13 number13 of13 charts13 and13 save13 the13 dashboard13

17 To13 re-shy‐run13 the13 dashboard13 select13 the13 Dashboard13 page13 and13 click13 the13 Load13 Saved13 Dashboard13 icon13

Page of 33 71 Page of 33 71

13

18 Select13 the13 saved13 dashboard13 from13 the13 list13

13

19 The13 dashboard13 runs13

13

Page of 34 71 Page of 34 71

Page of 35 71 Page of 35 71

Aler0ng13 with13 ELK13 ELK13 examines13 the13 log13 records13 in13 process13 to13 send13 alerts13 to13 third-shy‐party13 alerng13 event13 management13 and13 social13 media13 applicaons13 As13 shown13 previously13 in13 the13 ELK13 architecture13 the13 ELK13 strategy13 for13 alerng13 is13 to13 send13 alerts13 from13 the13 Logstash13 component13 directly13 Therefore13 log13 records13 and13 alerts13 are13 sent13 to13 both13 Elascsearch13 and13 third-shy‐party13 tools13 at13 the13 same13 instant13 using13 the13 output13 secon13 of13 the13 Logstash13 configuraon13 Any13 alerts13 are13 generated13 during13 the13 filter13 secon13 processing13 of13 the13 Logstash13 configuraon13

Create13 alerts13 in13 Logstash13 The13 Logstash13 filter13 secon13 is13 used13 to13 trigger13 alerts13 based13 on13 the13 log13 records13 being13 processed13 For13 example13 log13 records13 can13 be13 checked13 for13 errors13 warnings13 crashed13 or13 stopped13 messages13 and13 generate13 an13 alert13

Here13 is13 an13 example13 Logstash13 filter13 code13

13 if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

The13 specific13 tags13 are13 set13 against13 the13 log13 records13 to13 trigger13 the13 alert13 later13 in13 the13 configuraon13 in13 the13 output13 secon13

Control13 the13 number13 of13 generated13 alerts13 using13 the13 Logstash13 throcle13 plugin13 so13 that13 either13 a13 lot13 of13 instances13 in13 a13 fixed13 me13 period13 are13 required13 before13 an13 alert13 is13 sent13 or13 a13 specific13 number13 of13 instances13 triggers13 the13 alert13

Page of 36 71 Page of 36 71

Edit13 the13 plugin13 parameters13 to13 meet13 system13 requirements13

13 13 13 13 13 if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 10

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle13

after_count =gt 20

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 30

period =gt 1200

key =gt servicename_alarm_warning

Page of 37 71 Page of 37 71

add_tag =gt servicename_throttled

13 13 13 13 13 13 13 13 13

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 40

period =gt 1500

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

Enrich13 alerts13 with13 message13 detail13 Some13 of13 the13 third-shy‐party13 integraons13 benefit13 from13 having13 components13 of13 the13 raw13 log13 record13 sent13 in13 addion13 to13 the13 raw13 record13 or13 fields13 in13 the13 record13 edited13 transformed13 or13 translated13

Grok13 Logstash13 allows13 the13 log13 record13 to13 be13 broken13 up13 into13 logical13 secons13 and13 each13 secon13 stored13 in13 a13 new13 parameter13 using13 the13 grok13 parameter13 in13 the13 filter13 secon13 Here13 is13 an13 example13 grok13

13 grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

To13 build13 the13 grok13 statement13 Logstash13 offers13 an13 online13 tool13 at13 hcpsgrokdebugherokuappcom13

Translate13 If13 specific13 fields13 in13 the13 log13 record13 are13 coded13 (for13 example13 the13 Bluemix13 app13 name13 is13 oaen13 not13 sent13 in13 the13 syslog13 messages13 instead13 the13 GUID13 is13 sent13 which13 is13 not13 useful)13 then13 the13 filter13 secon13 translate13 plugin13 can13 be13 used13 to13 add13 new13 fields13 by13 holding13 the13 decoded13 versions13 of13 those13 values13

13 translate

field =gt data1

destination =gt data1

override =gt true

Page of 38 71 Page of 38 71

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

dc50f6e4-c052-436a-bf27-95ba392c4228 monverifyservicemanagement

0c677e82-5bfc-4e38-9155-e416249e3ee6 ms-catalog-postpyretic-pomiculture

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

However13 the13 translate13 plugin13 is13 not13 included13 with13 the13 Logstash13 installaon13 In13 the13 next13 secon13 I13 will13 13 describe13 how13 to13 add13 new13 plugins13 to13 the13 Logstash13 instance13

Add13 custom13 plugins13 to13 Logstash13 As13 an13 open13 source13 tool13 Logstash13 encourages13 users13 to13 develop13 content13 for13 all13 users13 to13 share13 Some13 of13 the13 plugins13 that13 were13 created13 by13 our13 community13 include13 a13 translaon13 plugin13 an13 output13 Slack13 plugin13 and13 an13 output13 syslog13 plugin13 Letrsquos13 look13 at13 how13 to13 add13 these13 plugins13 to13 your13 environment13

Filter13 translate13 plugin13 To13 add13 the13 translate13 plugin13

1 Download13 the13 plugin13

2 Copy13 the13 plugin13 file13 logstash-shy‐filter-shy‐translate-shy‐214gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-filter-translate 214

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Output13 Slack13 plugin13 To13 add13 the13 Slack13 plugin13

Page of 39 71 Page of 39 71

1 Download13 the13 plugin13 13

2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐slack-shy‐014gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-output-slack 014

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Output13 syslog13 plugin13 To13 add13 the13 syslog13 plugin13

1 Download13 the13 plugin13 13

2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐syslog-shy‐214gem13 to13 the13 ELK13 Virtual13 Server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-output-syslog 214

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Integrate13 Logstash13 with13 Slack13 To13 send13 generated13 alerts13 to13 Slack13 complete13 these13 steps13

1 Create13 a13 Slack13 integraon13 13

2 Use13 the13 Slack13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 Slack13 integraon13

113 Login13 to13 Slack13 and13 go13 to13 the13 Apps13 page13 13

Page of 40 71 Page of 40 71

13

313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13

13

A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13

13

Page of 42 71 Page of 42 71

The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13

13

The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13

13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

slack

url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5

channel =gt cloud-operations-mon

username =gt nick_cawood

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13

Page of 43 71 Page of 43 71

Page of 44 71 Page of 44 71

In13 Slack13 the13 alerts13 look13 like13 this13

13

Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13

1 Create13 a13 PagerDuty13 integraon13 13

2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13

1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13

13

2 Click13 Add13 New13 Service13

13

313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13

Page of 45 71 Page of 45 71

13

413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13

513 Click13 Add13 Service13

13

The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13

Page of 46 71 Page of 46 71

13

The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt be7eb1ea9c224018923e818d3e891d0b

incident_key =gt logstashhosttype

Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13

Page of 47 71 Page of 47 71

13

Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13

1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13

2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13

3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13

4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13

13 13 http

url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo

http_method =gt post

format =gt json

For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13

13 13 syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

Page of 48 71 Page of 48 71

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13

13

The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13

if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2

This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13

Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13

Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13

Logstash13 severity Shown13 in13 Omnibus13 as

alert Indeterminate

crical Warning

debug 7

emergency Clear

error Minor

informaonal 6

noce Crical

warning Major

Page of 49 71 Page of 49 71

13

Page of 50 71 Page of 50 71

Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13

input

tcp

port =gt 5000

type =gt syslog

filter

mutate

add_field =gt format =gt cf

grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

translate

field =gt data1

destination =gt data1

override =gt true

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

=== Check Alarm Criteria ===

Standard Key Words

if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

=== Alarm Exclusions ===

if [message] =~ (i)sometexthere

mutate remove_tag =gt servicename_alarm

Throttle alarms so we do not hit PD rate limits

Page of 52 71 Page of 52 71

if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 5

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle

after_count =gt 5

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 5

period =gt 1200

key =gt servicename_alarm_warning

add_tag =gt servicename_throttled

Page of 53 71 Page of 53 71

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

output

elasticsearch

hosts =gt localhost

stdout codec =gt rubydebug

if servicename_alarm_err in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

Page of 54 71 Page of 54 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 55 71 Page of 55 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

Page of 56 71 Page of 56 71

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert err message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_error in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

Page of 57 71 Page of 57 71

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 58 71 Page of 58 71

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 59 71 Page of 59 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert error message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

Page of 60 71 Page of 60 71

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

Page of 61 71 Page of 61 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 62 71 Page of 62 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt x

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert fatal message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

Page of 63 71 Page of 63 71

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

Page of 64 71 Page of 64 71

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 65 71 Page of 65 71

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt heavy_exclamation_mark

Page of 66 71 Page of 66 71

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert WARNING message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 67 71 Page of 67 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

Page of 68 71 Page of 68 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 69 71 Page of 69 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

Page of 70 71 Page of 70 71

Page of 71 71 Page of 71 71

Page 32: How to integrate the Elasticsearch Logstash Kibana (ELK ... · How to integrate the Elasticsearch Logstash Kibana (ELK) log analytics stack into IBM Bluemix Nick Cawood IBM Cloud

13

14 Add13 a13 tle13 and13 select13 whether13 to13 store13 the13 selected13 me13 frame13 with13 the13 dashboard13

13

15 Click13 Save13

16 You13 can13 edit13 the13 shape13 and13 number13 of13 charts13 and13 save13 the13 dashboard13

17 To13 re-shy‐run13 the13 dashboard13 select13 the13 Dashboard13 page13 and13 click13 the13 Load13 Saved13 Dashboard13 icon13

Page of 33 71 Page of 33 71

13

18 Select13 the13 saved13 dashboard13 from13 the13 list13

13

19 The13 dashboard13 runs13

13

Page of 34 71 Page of 34 71

Page of 35 71 Page of 35 71

Aler0ng13 with13 ELK13 ELK13 examines13 the13 log13 records13 in13 process13 to13 send13 alerts13 to13 third-shy‐party13 alerng13 event13 management13 and13 social13 media13 applicaons13 As13 shown13 previously13 in13 the13 ELK13 architecture13 the13 ELK13 strategy13 for13 alerng13 is13 to13 send13 alerts13 from13 the13 Logstash13 component13 directly13 Therefore13 log13 records13 and13 alerts13 are13 sent13 to13 both13 Elascsearch13 and13 third-shy‐party13 tools13 at13 the13 same13 instant13 using13 the13 output13 secon13 of13 the13 Logstash13 configuraon13 Any13 alerts13 are13 generated13 during13 the13 filter13 secon13 processing13 of13 the13 Logstash13 configuraon13

Create13 alerts13 in13 Logstash13 The13 Logstash13 filter13 secon13 is13 used13 to13 trigger13 alerts13 based13 on13 the13 log13 records13 being13 processed13 For13 example13 log13 records13 can13 be13 checked13 for13 errors13 warnings13 crashed13 or13 stopped13 messages13 and13 generate13 an13 alert13

Here13 is13 an13 example13 Logstash13 filter13 code13

13 if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

The13 specific13 tags13 are13 set13 against13 the13 log13 records13 to13 trigger13 the13 alert13 later13 in13 the13 configuraon13 in13 the13 output13 secon13

Control13 the13 number13 of13 generated13 alerts13 using13 the13 Logstash13 throcle13 plugin13 so13 that13 either13 a13 lot13 of13 instances13 in13 a13 fixed13 me13 period13 are13 required13 before13 an13 alert13 is13 sent13 or13 a13 specific13 number13 of13 instances13 triggers13 the13 alert13

Page of 36 71 Page of 36 71

Edit13 the13 plugin13 parameters13 to13 meet13 system13 requirements13

13 13 13 13 13 if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 10

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle13

after_count =gt 20

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 30

period =gt 1200

key =gt servicename_alarm_warning

Page of 37 71 Page of 37 71

add_tag =gt servicename_throttled

13 13 13 13 13 13 13 13 13

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 40

period =gt 1500

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

Enrich13 alerts13 with13 message13 detail13 Some13 of13 the13 third-shy‐party13 integraons13 benefit13 from13 having13 components13 of13 the13 raw13 log13 record13 sent13 in13 addion13 to13 the13 raw13 record13 or13 fields13 in13 the13 record13 edited13 transformed13 or13 translated13

Grok13 Logstash13 allows13 the13 log13 record13 to13 be13 broken13 up13 into13 logical13 secons13 and13 each13 secon13 stored13 in13 a13 new13 parameter13 using13 the13 grok13 parameter13 in13 the13 filter13 secon13 Here13 is13 an13 example13 grok13

13 grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

To13 build13 the13 grok13 statement13 Logstash13 offers13 an13 online13 tool13 at13 hcpsgrokdebugherokuappcom13

Translate13 If13 specific13 fields13 in13 the13 log13 record13 are13 coded13 (for13 example13 the13 Bluemix13 app13 name13 is13 oaen13 not13 sent13 in13 the13 syslog13 messages13 instead13 the13 GUID13 is13 sent13 which13 is13 not13 useful)13 then13 the13 filter13 secon13 translate13 plugin13 can13 be13 used13 to13 add13 new13 fields13 by13 holding13 the13 decoded13 versions13 of13 those13 values13

13 translate

field =gt data1

destination =gt data1

override =gt true

Page of 38 71 Page of 38 71

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

dc50f6e4-c052-436a-bf27-95ba392c4228 monverifyservicemanagement

0c677e82-5bfc-4e38-9155-e416249e3ee6 ms-catalog-postpyretic-pomiculture

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

However13 the13 translate13 plugin13 is13 not13 included13 with13 the13 Logstash13 installaon13 In13 the13 next13 secon13 I13 will13 13 describe13 how13 to13 add13 new13 plugins13 to13 the13 Logstash13 instance13

Add13 custom13 plugins13 to13 Logstash13 As13 an13 open13 source13 tool13 Logstash13 encourages13 users13 to13 develop13 content13 for13 all13 users13 to13 share13 Some13 of13 the13 plugins13 that13 were13 created13 by13 our13 community13 include13 a13 translaon13 plugin13 an13 output13 Slack13 plugin13 and13 an13 output13 syslog13 plugin13 Letrsquos13 look13 at13 how13 to13 add13 these13 plugins13 to13 your13 environment13

Filter13 translate13 plugin13 To13 add13 the13 translate13 plugin13

1 Download13 the13 plugin13

2 Copy13 the13 plugin13 file13 logstash-shy‐filter-shy‐translate-shy‐214gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-filter-translate 214

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Output13 Slack13 plugin13 To13 add13 the13 Slack13 plugin13

Page of 39 71 Page of 39 71

1 Download13 the13 plugin13 13

2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐slack-shy‐014gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-output-slack 014

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Output13 syslog13 plugin13 To13 add13 the13 syslog13 plugin13

1 Download13 the13 plugin13 13

2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐syslog-shy‐214gem13 to13 the13 ELK13 Virtual13 Server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-output-syslog 214

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Integrate13 Logstash13 with13 Slack13 To13 send13 generated13 alerts13 to13 Slack13 complete13 these13 steps13

1 Create13 a13 Slack13 integraon13 13

2 Use13 the13 Slack13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 Slack13 integraon13

113 Login13 to13 Slack13 and13 go13 to13 the13 Apps13 page13 13

Page of 40 71 Page of 40 71

13

313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13

13

A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13

13

Page of 42 71 Page of 42 71

The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13

13

The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13

13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

slack

url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5

channel =gt cloud-operations-mon

username =gt nick_cawood

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13

Page of 43 71 Page of 43 71

Page of 44 71 Page of 44 71

In13 Slack13 the13 alerts13 look13 like13 this13

13

Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13

1 Create13 a13 PagerDuty13 integraon13 13

2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13

1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13

13

2 Click13 Add13 New13 Service13

13

313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13

Page of 45 71 Page of 45 71

13

413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13

513 Click13 Add13 Service13

13

The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13

Page of 46 71 Page of 46 71

13

The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt be7eb1ea9c224018923e818d3e891d0b

incident_key =gt logstashhosttype

Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13

Page of 47 71 Page of 47 71

13

Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13

1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13

2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13

3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13

4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13

13 13 http

url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo

http_method =gt post

format =gt json

For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13

13 13 syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

Page of 48 71 Page of 48 71

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13

13

The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13

if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2

This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13

Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13

Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13

Logstash13 severity Shown13 in13 Omnibus13 as

alert Indeterminate

crical Warning

debug 7

emergency Clear

error Minor

informaonal 6

noce Crical

warning Major

Page of 49 71 Page of 49 71

13

Page of 50 71 Page of 50 71

Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13

input

tcp

port =gt 5000

type =gt syslog

filter

mutate

add_field =gt format =gt cf

grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

translate

field =gt data1

destination =gt data1

override =gt true

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

=== Check Alarm Criteria ===

Standard Key Words

if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

=== Alarm Exclusions ===

if [message] =~ (i)sometexthere

mutate remove_tag =gt servicename_alarm

Throttle alarms so we do not hit PD rate limits

Page of 52 71 Page of 52 71

if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 5

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle

after_count =gt 5

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 5

period =gt 1200

key =gt servicename_alarm_warning

add_tag =gt servicename_throttled

Page of 53 71 Page of 53 71

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

output

elasticsearch

hosts =gt localhost

stdout codec =gt rubydebug

if servicename_alarm_err in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

Page of 54 71 Page of 54 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 55 71 Page of 55 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

Page of 56 71 Page of 56 71

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert err message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_error in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

Page of 57 71 Page of 57 71

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 58 71 Page of 58 71

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 59 71 Page of 59 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert error message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

Page of 60 71 Page of 60 71

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

Page of 61 71 Page of 61 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 62 71 Page of 62 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt x

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert fatal message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

Page of 63 71 Page of 63 71

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

Page of 64 71 Page of 64 71

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 65 71 Page of 65 71

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt heavy_exclamation_mark

Page of 66 71 Page of 66 71

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert WARNING message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 67 71 Page of 67 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

Page of 68 71 Page of 68 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 69 71 Page of 69 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

Page of 70 71 Page of 70 71

Page of 71 71 Page of 71 71

Page 33: How to integrate the Elasticsearch Logstash Kibana (ELK ... · How to integrate the Elasticsearch Logstash Kibana (ELK) log analytics stack into IBM Bluemix Nick Cawood IBM Cloud

13

18 Select13 the13 saved13 dashboard13 from13 the13 list13

13

19 The13 dashboard13 runs13

13

Page of 34 71 Page of 34 71

Page of 35 71 Page of 35 71

Aler0ng13 with13 ELK13 ELK13 examines13 the13 log13 records13 in13 process13 to13 send13 alerts13 to13 third-shy‐party13 alerng13 event13 management13 and13 social13 media13 applicaons13 As13 shown13 previously13 in13 the13 ELK13 architecture13 the13 ELK13 strategy13 for13 alerng13 is13 to13 send13 alerts13 from13 the13 Logstash13 component13 directly13 Therefore13 log13 records13 and13 alerts13 are13 sent13 to13 both13 Elascsearch13 and13 third-shy‐party13 tools13 at13 the13 same13 instant13 using13 the13 output13 secon13 of13 the13 Logstash13 configuraon13 Any13 alerts13 are13 generated13 during13 the13 filter13 secon13 processing13 of13 the13 Logstash13 configuraon13

Create13 alerts13 in13 Logstash13 The13 Logstash13 filter13 secon13 is13 used13 to13 trigger13 alerts13 based13 on13 the13 log13 records13 being13 processed13 For13 example13 log13 records13 can13 be13 checked13 for13 errors13 warnings13 crashed13 or13 stopped13 messages13 and13 generate13 an13 alert13

Here13 is13 an13 example13 Logstash13 filter13 code13

13 if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

The13 specific13 tags13 are13 set13 against13 the13 log13 records13 to13 trigger13 the13 alert13 later13 in13 the13 configuraon13 in13 the13 output13 secon13

Control13 the13 number13 of13 generated13 alerts13 using13 the13 Logstash13 throcle13 plugin13 so13 that13 either13 a13 lot13 of13 instances13 in13 a13 fixed13 me13 period13 are13 required13 before13 an13 alert13 is13 sent13 or13 a13 specific13 number13 of13 instances13 triggers13 the13 alert13

Page of 36 71 Page of 36 71

Edit13 the13 plugin13 parameters13 to13 meet13 system13 requirements13

13 13 13 13 13 if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 10

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle13

after_count =gt 20

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 30

period =gt 1200

key =gt servicename_alarm_warning

Page of 37 71 Page of 37 71

add_tag =gt servicename_throttled

13 13 13 13 13 13 13 13 13

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 40

period =gt 1500

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

Enrich13 alerts13 with13 message13 detail13 Some13 of13 the13 third-shy‐party13 integraons13 benefit13 from13 having13 components13 of13 the13 raw13 log13 record13 sent13 in13 addion13 to13 the13 raw13 record13 or13 fields13 in13 the13 record13 edited13 transformed13 or13 translated13

Grok13 Logstash13 allows13 the13 log13 record13 to13 be13 broken13 up13 into13 logical13 secons13 and13 each13 secon13 stored13 in13 a13 new13 parameter13 using13 the13 grok13 parameter13 in13 the13 filter13 secon13 Here13 is13 an13 example13 grok13

13 grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

To13 build13 the13 grok13 statement13 Logstash13 offers13 an13 online13 tool13 at13 hcpsgrokdebugherokuappcom13

Translate13 If13 specific13 fields13 in13 the13 log13 record13 are13 coded13 (for13 example13 the13 Bluemix13 app13 name13 is13 oaen13 not13 sent13 in13 the13 syslog13 messages13 instead13 the13 GUID13 is13 sent13 which13 is13 not13 useful)13 then13 the13 filter13 secon13 translate13 plugin13 can13 be13 used13 to13 add13 new13 fields13 by13 holding13 the13 decoded13 versions13 of13 those13 values13

13 translate

field =gt data1

destination =gt data1

override =gt true

Page of 38 71 Page of 38 71

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

dc50f6e4-c052-436a-bf27-95ba392c4228 monverifyservicemanagement

0c677e82-5bfc-4e38-9155-e416249e3ee6 ms-catalog-postpyretic-pomiculture

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

However13 the13 translate13 plugin13 is13 not13 included13 with13 the13 Logstash13 installaon13 In13 the13 next13 secon13 I13 will13 13 describe13 how13 to13 add13 new13 plugins13 to13 the13 Logstash13 instance13

Add13 custom13 plugins13 to13 Logstash13 As13 an13 open13 source13 tool13 Logstash13 encourages13 users13 to13 develop13 content13 for13 all13 users13 to13 share13 Some13 of13 the13 plugins13 that13 were13 created13 by13 our13 community13 include13 a13 translaon13 plugin13 an13 output13 Slack13 plugin13 and13 an13 output13 syslog13 plugin13 Letrsquos13 look13 at13 how13 to13 add13 these13 plugins13 to13 your13 environment13

Filter13 translate13 plugin13 To13 add13 the13 translate13 plugin13

1 Download13 the13 plugin13

2 Copy13 the13 plugin13 file13 logstash-shy‐filter-shy‐translate-shy‐214gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-filter-translate 214

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Output13 Slack13 plugin13 To13 add13 the13 Slack13 plugin13

Page of 39 71 Page of 39 71

1 Download13 the13 plugin13 13

2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐slack-shy‐014gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-output-slack 014

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Output13 syslog13 plugin13 To13 add13 the13 syslog13 plugin13

1 Download13 the13 plugin13 13

2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐syslog-shy‐214gem13 to13 the13 ELK13 Virtual13 Server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-output-syslog 214

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Integrate13 Logstash13 with13 Slack13 To13 send13 generated13 alerts13 to13 Slack13 complete13 these13 steps13

1 Create13 a13 Slack13 integraon13 13

2 Use13 the13 Slack13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 Slack13 integraon13

113 Login13 to13 Slack13 and13 go13 to13 the13 Apps13 page13 13

Page of 40 71 Page of 40 71

13

313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13

13

A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13

13

Page of 42 71 Page of 42 71

The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13

13

The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13

13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

slack

url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5

channel =gt cloud-operations-mon

username =gt nick_cawood

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13

Page of 43 71 Page of 43 71

Page of 44 71 Page of 44 71

In13 Slack13 the13 alerts13 look13 like13 this13

13

Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13

1 Create13 a13 PagerDuty13 integraon13 13

2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13

1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13

13

2 Click13 Add13 New13 Service13

13

313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13

Page of 45 71 Page of 45 71

13

413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13

513 Click13 Add13 Service13

13

The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13

Page of 46 71 Page of 46 71

13

The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt be7eb1ea9c224018923e818d3e891d0b

incident_key =gt logstashhosttype

Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13

Page of 47 71 Page of 47 71

13

Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13

1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13

2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13

3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13

4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13

13 13 http

url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo

http_method =gt post

format =gt json

For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13

13 13 syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

Page of 48 71 Page of 48 71

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13

13

The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13

if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2

This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13

Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13

Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13

Logstash13 severity Shown13 in13 Omnibus13 as

alert Indeterminate

crical Warning

debug 7

emergency Clear

error Minor

informaonal 6

noce Crical

warning Major

Page of 49 71 Page of 49 71

13

Page of 50 71 Page of 50 71

Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13

input

tcp

port =gt 5000

type =gt syslog

filter

mutate

add_field =gt format =gt cf

grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

translate

field =gt data1

destination =gt data1

override =gt true

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

=== Check Alarm Criteria ===

Standard Key Words

if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

=== Alarm Exclusions ===

if [message] =~ (i)sometexthere

mutate remove_tag =gt servicename_alarm

Throttle alarms so we do not hit PD rate limits

Page of 52 71 Page of 52 71

if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 5

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle

after_count =gt 5

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 5

period =gt 1200

key =gt servicename_alarm_warning

add_tag =gt servicename_throttled

Page of 53 71 Page of 53 71

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

output

elasticsearch

hosts =gt localhost

stdout codec =gt rubydebug

if servicename_alarm_err in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

Page of 54 71 Page of 54 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 55 71 Page of 55 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

Page of 56 71 Page of 56 71

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert err message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_error in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

Page of 57 71 Page of 57 71

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 58 71 Page of 58 71

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 59 71 Page of 59 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert error message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

Page of 60 71 Page of 60 71

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

Page of 61 71 Page of 61 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 62 71 Page of 62 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt x

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert fatal message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

Page of 63 71 Page of 63 71

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

Page of 64 71 Page of 64 71

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 65 71 Page of 65 71

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt heavy_exclamation_mark

Page of 66 71 Page of 66 71

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert WARNING message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 67 71 Page of 67 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

Page of 68 71 Page of 68 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 69 71 Page of 69 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

Page of 70 71 Page of 70 71

Page of 71 71 Page of 71 71

Page 34: How to integrate the Elasticsearch Logstash Kibana (ELK ... · How to integrate the Elasticsearch Logstash Kibana (ELK) log analytics stack into IBM Bluemix Nick Cawood IBM Cloud

Page of 35 71 Page of 35 71

Aler0ng13 with13 ELK13 ELK13 examines13 the13 log13 records13 in13 process13 to13 send13 alerts13 to13 third-shy‐party13 alerng13 event13 management13 and13 social13 media13 applicaons13 As13 shown13 previously13 in13 the13 ELK13 architecture13 the13 ELK13 strategy13 for13 alerng13 is13 to13 send13 alerts13 from13 the13 Logstash13 component13 directly13 Therefore13 log13 records13 and13 alerts13 are13 sent13 to13 both13 Elascsearch13 and13 third-shy‐party13 tools13 at13 the13 same13 instant13 using13 the13 output13 secon13 of13 the13 Logstash13 configuraon13 Any13 alerts13 are13 generated13 during13 the13 filter13 secon13 processing13 of13 the13 Logstash13 configuraon13

Create13 alerts13 in13 Logstash13 The13 Logstash13 filter13 secon13 is13 used13 to13 trigger13 alerts13 based13 on13 the13 log13 records13 being13 processed13 For13 example13 log13 records13 can13 be13 checked13 for13 errors13 warnings13 crashed13 or13 stopped13 messages13 and13 generate13 an13 alert13

Here13 is13 an13 example13 Logstash13 filter13 code13

13 if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

The13 specific13 tags13 are13 set13 against13 the13 log13 records13 to13 trigger13 the13 alert13 later13 in13 the13 configuraon13 in13 the13 output13 secon13

Control13 the13 number13 of13 generated13 alerts13 using13 the13 Logstash13 throcle13 plugin13 so13 that13 either13 a13 lot13 of13 instances13 in13 a13 fixed13 me13 period13 are13 required13 before13 an13 alert13 is13 sent13 or13 a13 specific13 number13 of13 instances13 triggers13 the13 alert13

Page of 36 71 Page of 36 71

Edit13 the13 plugin13 parameters13 to13 meet13 system13 requirements13

13 13 13 13 13 if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 10

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle13

after_count =gt 20

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 30

period =gt 1200

key =gt servicename_alarm_warning

Page of 37 71 Page of 37 71

add_tag =gt servicename_throttled

13 13 13 13 13 13 13 13 13

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 40

period =gt 1500

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

Enrich13 alerts13 with13 message13 detail13 Some13 of13 the13 third-shy‐party13 integraons13 benefit13 from13 having13 components13 of13 the13 raw13 log13 record13 sent13 in13 addion13 to13 the13 raw13 record13 or13 fields13 in13 the13 record13 edited13 transformed13 or13 translated13

Grok13 Logstash13 allows13 the13 log13 record13 to13 be13 broken13 up13 into13 logical13 secons13 and13 each13 secon13 stored13 in13 a13 new13 parameter13 using13 the13 grok13 parameter13 in13 the13 filter13 secon13 Here13 is13 an13 example13 grok13

13 grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

To13 build13 the13 grok13 statement13 Logstash13 offers13 an13 online13 tool13 at13 hcpsgrokdebugherokuappcom13

Translate13 If13 specific13 fields13 in13 the13 log13 record13 are13 coded13 (for13 example13 the13 Bluemix13 app13 name13 is13 oaen13 not13 sent13 in13 the13 syslog13 messages13 instead13 the13 GUID13 is13 sent13 which13 is13 not13 useful)13 then13 the13 filter13 secon13 translate13 plugin13 can13 be13 used13 to13 add13 new13 fields13 by13 holding13 the13 decoded13 versions13 of13 those13 values13

13 translate

field =gt data1

destination =gt data1

override =gt true

Page of 38 71 Page of 38 71

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

dc50f6e4-c052-436a-bf27-95ba392c4228 monverifyservicemanagement

0c677e82-5bfc-4e38-9155-e416249e3ee6 ms-catalog-postpyretic-pomiculture

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

However13 the13 translate13 plugin13 is13 not13 included13 with13 the13 Logstash13 installaon13 In13 the13 next13 secon13 I13 will13 13 describe13 how13 to13 add13 new13 plugins13 to13 the13 Logstash13 instance13

Add13 custom13 plugins13 to13 Logstash13 As13 an13 open13 source13 tool13 Logstash13 encourages13 users13 to13 develop13 content13 for13 all13 users13 to13 share13 Some13 of13 the13 plugins13 that13 were13 created13 by13 our13 community13 include13 a13 translaon13 plugin13 an13 output13 Slack13 plugin13 and13 an13 output13 syslog13 plugin13 Letrsquos13 look13 at13 how13 to13 add13 these13 plugins13 to13 your13 environment13

Filter13 translate13 plugin13 To13 add13 the13 translate13 plugin13

1 Download13 the13 plugin13

2 Copy13 the13 plugin13 file13 logstash-shy‐filter-shy‐translate-shy‐214gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-filter-translate 214

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Output13 Slack13 plugin13 To13 add13 the13 Slack13 plugin13

Page of 39 71 Page of 39 71

1 Download13 the13 plugin13 13

2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐slack-shy‐014gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-output-slack 014

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Output13 syslog13 plugin13 To13 add13 the13 syslog13 plugin13

1 Download13 the13 plugin13 13

2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐syslog-shy‐214gem13 to13 the13 ELK13 Virtual13 Server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-output-syslog 214

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Integrate13 Logstash13 with13 Slack13 To13 send13 generated13 alerts13 to13 Slack13 complete13 these13 steps13

1 Create13 a13 Slack13 integraon13 13

2 Use13 the13 Slack13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 Slack13 integraon13

113 Login13 to13 Slack13 and13 go13 to13 the13 Apps13 page13 13

Page of 40 71 Page of 40 71

13

313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13

13

A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13

13

Page of 42 71 Page of 42 71

The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13

13

The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13

13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

slack

url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5

channel =gt cloud-operations-mon

username =gt nick_cawood

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13

Page of 43 71 Page of 43 71

Page of 44 71 Page of 44 71

In13 Slack13 the13 alerts13 look13 like13 this13

13

Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13

1 Create13 a13 PagerDuty13 integraon13 13

2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13

1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13

13

2 Click13 Add13 New13 Service13

13

313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13

Page of 45 71 Page of 45 71

13

413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13

513 Click13 Add13 Service13

13

The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13

Page of 46 71 Page of 46 71

13

The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt be7eb1ea9c224018923e818d3e891d0b

incident_key =gt logstashhosttype

Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13

Page of 47 71 Page of 47 71

13

Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13

1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13

2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13

3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13

4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13

13 13 http

url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo

http_method =gt post

format =gt json

For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13

13 13 syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

Page of 48 71 Page of 48 71

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13

13

The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13

if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2

This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13

Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13

Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13

Logstash13 severity Shown13 in13 Omnibus13 as

alert Indeterminate

crical Warning

debug 7

emergency Clear

error Minor

informaonal 6

noce Crical

warning Major

Page of 49 71 Page of 49 71

13

Page of 50 71 Page of 50 71

Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13

input

tcp

port =gt 5000

type =gt syslog

filter

mutate

add_field =gt format =gt cf

grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

translate

field =gt data1

destination =gt data1

override =gt true

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

=== Check Alarm Criteria ===

Standard Key Words

if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

=== Alarm Exclusions ===

if [message] =~ (i)sometexthere

mutate remove_tag =gt servicename_alarm

Throttle alarms so we do not hit PD rate limits

Page of 52 71 Page of 52 71

if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 5

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle

after_count =gt 5

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 5

period =gt 1200

key =gt servicename_alarm_warning

add_tag =gt servicename_throttled

Page of 53 71 Page of 53 71

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

output

elasticsearch

hosts =gt localhost

stdout codec =gt rubydebug

if servicename_alarm_err in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

Page of 54 71 Page of 54 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 55 71 Page of 55 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

Page of 56 71 Page of 56 71

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert err message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_error in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

Page of 57 71 Page of 57 71

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 58 71 Page of 58 71

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 59 71 Page of 59 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert error message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

Page of 60 71 Page of 60 71

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

Page of 61 71 Page of 61 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 62 71 Page of 62 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt x

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert fatal message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

Page of 63 71 Page of 63 71

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

Page of 64 71 Page of 64 71

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 65 71 Page of 65 71

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt heavy_exclamation_mark

Page of 66 71 Page of 66 71

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert WARNING message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 67 71 Page of 67 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

Page of 68 71 Page of 68 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 69 71 Page of 69 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

Page of 70 71 Page of 70 71

Page of 71 71 Page of 71 71

Page 35: How to integrate the Elasticsearch Logstash Kibana (ELK ... · How to integrate the Elasticsearch Logstash Kibana (ELK) log analytics stack into IBM Bluemix Nick Cawood IBM Cloud

Aler0ng13 with13 ELK13 ELK13 examines13 the13 log13 records13 in13 process13 to13 send13 alerts13 to13 third-shy‐party13 alerng13 event13 management13 and13 social13 media13 applicaons13 As13 shown13 previously13 in13 the13 ELK13 architecture13 the13 ELK13 strategy13 for13 alerng13 is13 to13 send13 alerts13 from13 the13 Logstash13 component13 directly13 Therefore13 log13 records13 and13 alerts13 are13 sent13 to13 both13 Elascsearch13 and13 third-shy‐party13 tools13 at13 the13 same13 instant13 using13 the13 output13 secon13 of13 the13 Logstash13 configuraon13 Any13 alerts13 are13 generated13 during13 the13 filter13 secon13 processing13 of13 the13 Logstash13 configuraon13

Create13 alerts13 in13 Logstash13 The13 Logstash13 filter13 secon13 is13 used13 to13 trigger13 alerts13 based13 on13 the13 log13 records13 being13 processed13 For13 example13 log13 records13 can13 be13 checked13 for13 errors13 warnings13 crashed13 or13 stopped13 messages13 and13 generate13 an13 alert13

Here13 is13 an13 example13 Logstash13 filter13 code13

13 if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

The13 specific13 tags13 are13 set13 against13 the13 log13 records13 to13 trigger13 the13 alert13 later13 in13 the13 configuraon13 in13 the13 output13 secon13

Control13 the13 number13 of13 generated13 alerts13 using13 the13 Logstash13 throcle13 plugin13 so13 that13 either13 a13 lot13 of13 instances13 in13 a13 fixed13 me13 period13 are13 required13 before13 an13 alert13 is13 sent13 or13 a13 specific13 number13 of13 instances13 triggers13 the13 alert13

Page of 36 71 Page of 36 71

Edit13 the13 plugin13 parameters13 to13 meet13 system13 requirements13

13 13 13 13 13 if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 10

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle13

after_count =gt 20

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 30

period =gt 1200

key =gt servicename_alarm_warning

Page of 37 71 Page of 37 71

add_tag =gt servicename_throttled

13 13 13 13 13 13 13 13 13

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 40

period =gt 1500

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

Enrich13 alerts13 with13 message13 detail13 Some13 of13 the13 third-shy‐party13 integraons13 benefit13 from13 having13 components13 of13 the13 raw13 log13 record13 sent13 in13 addion13 to13 the13 raw13 record13 or13 fields13 in13 the13 record13 edited13 transformed13 or13 translated13

Grok13 Logstash13 allows13 the13 log13 record13 to13 be13 broken13 up13 into13 logical13 secons13 and13 each13 secon13 stored13 in13 a13 new13 parameter13 using13 the13 grok13 parameter13 in13 the13 filter13 secon13 Here13 is13 an13 example13 grok13

13 grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

To13 build13 the13 grok13 statement13 Logstash13 offers13 an13 online13 tool13 at13 hcpsgrokdebugherokuappcom13

Translate13 If13 specific13 fields13 in13 the13 log13 record13 are13 coded13 (for13 example13 the13 Bluemix13 app13 name13 is13 oaen13 not13 sent13 in13 the13 syslog13 messages13 instead13 the13 GUID13 is13 sent13 which13 is13 not13 useful)13 then13 the13 filter13 secon13 translate13 plugin13 can13 be13 used13 to13 add13 new13 fields13 by13 holding13 the13 decoded13 versions13 of13 those13 values13

13 translate

field =gt data1

destination =gt data1

override =gt true

Page of 38 71 Page of 38 71

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

dc50f6e4-c052-436a-bf27-95ba392c4228 monverifyservicemanagement

0c677e82-5bfc-4e38-9155-e416249e3ee6 ms-catalog-postpyretic-pomiculture

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

However13 the13 translate13 plugin13 is13 not13 included13 with13 the13 Logstash13 installaon13 In13 the13 next13 secon13 I13 will13 13 describe13 how13 to13 add13 new13 plugins13 to13 the13 Logstash13 instance13

Add13 custom13 plugins13 to13 Logstash13 As13 an13 open13 source13 tool13 Logstash13 encourages13 users13 to13 develop13 content13 for13 all13 users13 to13 share13 Some13 of13 the13 plugins13 that13 were13 created13 by13 our13 community13 include13 a13 translaon13 plugin13 an13 output13 Slack13 plugin13 and13 an13 output13 syslog13 plugin13 Letrsquos13 look13 at13 how13 to13 add13 these13 plugins13 to13 your13 environment13

Filter13 translate13 plugin13 To13 add13 the13 translate13 plugin13

1 Download13 the13 plugin13

2 Copy13 the13 plugin13 file13 logstash-shy‐filter-shy‐translate-shy‐214gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-filter-translate 214

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Output13 Slack13 plugin13 To13 add13 the13 Slack13 plugin13

Page of 39 71 Page of 39 71

1 Download13 the13 plugin13 13

2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐slack-shy‐014gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-output-slack 014

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Output13 syslog13 plugin13 To13 add13 the13 syslog13 plugin13

1 Download13 the13 plugin13 13

2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐syslog-shy‐214gem13 to13 the13 ELK13 Virtual13 Server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-output-syslog 214

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Integrate13 Logstash13 with13 Slack13 To13 send13 generated13 alerts13 to13 Slack13 complete13 these13 steps13

1 Create13 a13 Slack13 integraon13 13

2 Use13 the13 Slack13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 Slack13 integraon13

113 Login13 to13 Slack13 and13 go13 to13 the13 Apps13 page13 13

Page of 40 71 Page of 40 71

13

313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13

13

A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13

13

Page of 42 71 Page of 42 71

The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13

13

The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13

13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

slack

url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5

channel =gt cloud-operations-mon

username =gt nick_cawood

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13

Page of 43 71 Page of 43 71

Page of 44 71 Page of 44 71

In13 Slack13 the13 alerts13 look13 like13 this13

13

Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13

1 Create13 a13 PagerDuty13 integraon13 13

2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13

1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13

13

2 Click13 Add13 New13 Service13

13

313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13

Page of 45 71 Page of 45 71

13

413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13

513 Click13 Add13 Service13

13

The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13

Page of 46 71 Page of 46 71

13

The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt be7eb1ea9c224018923e818d3e891d0b

incident_key =gt logstashhosttype

Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13

Page of 47 71 Page of 47 71

13

Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13

1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13

2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13

3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13

4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13

13 13 http

url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo

http_method =gt post

format =gt json

For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13

13 13 syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

Page of 48 71 Page of 48 71

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13

13

The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13

if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2

This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13

Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13

Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13

Logstash13 severity Shown13 in13 Omnibus13 as

alert Indeterminate

crical Warning

debug 7

emergency Clear

error Minor

informaonal 6

noce Crical

warning Major

Page of 49 71 Page of 49 71

13

Page of 50 71 Page of 50 71

Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13

input

tcp

port =gt 5000

type =gt syslog

filter

mutate

add_field =gt format =gt cf

grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

translate

field =gt data1

destination =gt data1

override =gt true

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

=== Check Alarm Criteria ===

Standard Key Words

if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

=== Alarm Exclusions ===

if [message] =~ (i)sometexthere

mutate remove_tag =gt servicename_alarm

Throttle alarms so we do not hit PD rate limits

Page of 52 71 Page of 52 71

if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 5

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle

after_count =gt 5

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 5

period =gt 1200

key =gt servicename_alarm_warning

add_tag =gt servicename_throttled

Page of 53 71 Page of 53 71

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

output

elasticsearch

hosts =gt localhost

stdout codec =gt rubydebug

if servicename_alarm_err in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

Page of 54 71 Page of 54 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 55 71 Page of 55 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

Page of 56 71 Page of 56 71

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert err message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_error in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

Page of 57 71 Page of 57 71

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 58 71 Page of 58 71

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 59 71 Page of 59 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert error message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

Page of 60 71 Page of 60 71

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

Page of 61 71 Page of 61 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 62 71 Page of 62 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt x

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert fatal message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

Page of 63 71 Page of 63 71

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

Page of 64 71 Page of 64 71

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 65 71 Page of 65 71

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt heavy_exclamation_mark

Page of 66 71 Page of 66 71

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert WARNING message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 67 71 Page of 67 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

Page of 68 71 Page of 68 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 69 71 Page of 69 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

Page of 70 71 Page of 70 71

Page of 71 71 Page of 71 71

Page 36: How to integrate the Elasticsearch Logstash Kibana (ELK ... · How to integrate the Elasticsearch Logstash Kibana (ELK) log analytics stack into IBM Bluemix Nick Cawood IBM Cloud

Edit13 the13 plugin13 parameters13 to13 meet13 system13 requirements13

13 13 13 13 13 if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 10

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle13

after_count =gt 20

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 30

period =gt 1200

key =gt servicename_alarm_warning

Page of 37 71 Page of 37 71

add_tag =gt servicename_throttled

13 13 13 13 13 13 13 13 13

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 40

period =gt 1500

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

Enrich13 alerts13 with13 message13 detail13 Some13 of13 the13 third-shy‐party13 integraons13 benefit13 from13 having13 components13 of13 the13 raw13 log13 record13 sent13 in13 addion13 to13 the13 raw13 record13 or13 fields13 in13 the13 record13 edited13 transformed13 or13 translated13

Grok13 Logstash13 allows13 the13 log13 record13 to13 be13 broken13 up13 into13 logical13 secons13 and13 each13 secon13 stored13 in13 a13 new13 parameter13 using13 the13 grok13 parameter13 in13 the13 filter13 secon13 Here13 is13 an13 example13 grok13

13 grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

To13 build13 the13 grok13 statement13 Logstash13 offers13 an13 online13 tool13 at13 hcpsgrokdebugherokuappcom13

Translate13 If13 specific13 fields13 in13 the13 log13 record13 are13 coded13 (for13 example13 the13 Bluemix13 app13 name13 is13 oaen13 not13 sent13 in13 the13 syslog13 messages13 instead13 the13 GUID13 is13 sent13 which13 is13 not13 useful)13 then13 the13 filter13 secon13 translate13 plugin13 can13 be13 used13 to13 add13 new13 fields13 by13 holding13 the13 decoded13 versions13 of13 those13 values13

13 translate

field =gt data1

destination =gt data1

override =gt true

Page of 38 71 Page of 38 71

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

dc50f6e4-c052-436a-bf27-95ba392c4228 monverifyservicemanagement

0c677e82-5bfc-4e38-9155-e416249e3ee6 ms-catalog-postpyretic-pomiculture

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

However13 the13 translate13 plugin13 is13 not13 included13 with13 the13 Logstash13 installaon13 In13 the13 next13 secon13 I13 will13 13 describe13 how13 to13 add13 new13 plugins13 to13 the13 Logstash13 instance13

Add13 custom13 plugins13 to13 Logstash13 As13 an13 open13 source13 tool13 Logstash13 encourages13 users13 to13 develop13 content13 for13 all13 users13 to13 share13 Some13 of13 the13 plugins13 that13 were13 created13 by13 our13 community13 include13 a13 translaon13 plugin13 an13 output13 Slack13 plugin13 and13 an13 output13 syslog13 plugin13 Letrsquos13 look13 at13 how13 to13 add13 these13 plugins13 to13 your13 environment13

Filter13 translate13 plugin13 To13 add13 the13 translate13 plugin13

1 Download13 the13 plugin13

2 Copy13 the13 plugin13 file13 logstash-shy‐filter-shy‐translate-shy‐214gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-filter-translate 214

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Output13 Slack13 plugin13 To13 add13 the13 Slack13 plugin13

Page of 39 71 Page of 39 71

1 Download13 the13 plugin13 13

2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐slack-shy‐014gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-output-slack 014

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Output13 syslog13 plugin13 To13 add13 the13 syslog13 plugin13

1 Download13 the13 plugin13 13

2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐syslog-shy‐214gem13 to13 the13 ELK13 Virtual13 Server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-output-syslog 214

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Integrate13 Logstash13 with13 Slack13 To13 send13 generated13 alerts13 to13 Slack13 complete13 these13 steps13

1 Create13 a13 Slack13 integraon13 13

2 Use13 the13 Slack13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 Slack13 integraon13

113 Login13 to13 Slack13 and13 go13 to13 the13 Apps13 page13 13

Page of 40 71 Page of 40 71

13

313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13

13

A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13

13

Page of 42 71 Page of 42 71

The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13

13

The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13

13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

slack

url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5

channel =gt cloud-operations-mon

username =gt nick_cawood

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13

Page of 43 71 Page of 43 71

Page of 44 71 Page of 44 71

In13 Slack13 the13 alerts13 look13 like13 this13

13

Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13

1 Create13 a13 PagerDuty13 integraon13 13

2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13

1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13

13

2 Click13 Add13 New13 Service13

13

313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13

Page of 45 71 Page of 45 71

13

413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13

513 Click13 Add13 Service13

13

The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13

Page of 46 71 Page of 46 71

13

The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt be7eb1ea9c224018923e818d3e891d0b

incident_key =gt logstashhosttype

Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13

Page of 47 71 Page of 47 71

13

Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13

1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13

2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13

3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13

4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13

13 13 http

url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo

http_method =gt post

format =gt json

For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13

13 13 syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

Page of 48 71 Page of 48 71

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13

13

The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13

if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2

This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13

Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13

Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13

Logstash13 severity Shown13 in13 Omnibus13 as

alert Indeterminate

crical Warning

debug 7

emergency Clear

error Minor

informaonal 6

noce Crical

warning Major

Page of 49 71 Page of 49 71

13

Page of 50 71 Page of 50 71

Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13

input

tcp

port =gt 5000

type =gt syslog

filter

mutate

add_field =gt format =gt cf

grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

translate

field =gt data1

destination =gt data1

override =gt true

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

=== Check Alarm Criteria ===

Standard Key Words

if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

=== Alarm Exclusions ===

if [message] =~ (i)sometexthere

mutate remove_tag =gt servicename_alarm

Throttle alarms so we do not hit PD rate limits

Page of 52 71 Page of 52 71

if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 5

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle

after_count =gt 5

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 5

period =gt 1200

key =gt servicename_alarm_warning

add_tag =gt servicename_throttled

Page of 53 71 Page of 53 71

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

output

elasticsearch

hosts =gt localhost

stdout codec =gt rubydebug

if servicename_alarm_err in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

Page of 54 71 Page of 54 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 55 71 Page of 55 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

Page of 56 71 Page of 56 71

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert err message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_error in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

Page of 57 71 Page of 57 71

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 58 71 Page of 58 71

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 59 71 Page of 59 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert error message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

Page of 60 71 Page of 60 71

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

Page of 61 71 Page of 61 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 62 71 Page of 62 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt x

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert fatal message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

Page of 63 71 Page of 63 71

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

Page of 64 71 Page of 64 71

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 65 71 Page of 65 71

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt heavy_exclamation_mark

Page of 66 71 Page of 66 71

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert WARNING message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 67 71 Page of 67 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

Page of 68 71 Page of 68 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 69 71 Page of 69 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

Page of 70 71 Page of 70 71

Page of 71 71 Page of 71 71

Page 37: How to integrate the Elasticsearch Logstash Kibana (ELK ... · How to integrate the Elasticsearch Logstash Kibana (ELK) log analytics stack into IBM Bluemix Nick Cawood IBM Cloud

add_tag =gt servicename_throttled

13 13 13 13 13 13 13 13 13

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 40

period =gt 1500

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

Enrich13 alerts13 with13 message13 detail13 Some13 of13 the13 third-shy‐party13 integraons13 benefit13 from13 having13 components13 of13 the13 raw13 log13 record13 sent13 in13 addion13 to13 the13 raw13 record13 or13 fields13 in13 the13 record13 edited13 transformed13 or13 translated13

Grok13 Logstash13 allows13 the13 log13 record13 to13 be13 broken13 up13 into13 logical13 secons13 and13 each13 secon13 stored13 in13 a13 new13 parameter13 using13 the13 grok13 parameter13 in13 the13 filter13 secon13 Here13 is13 an13 example13 grok13

13 grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

To13 build13 the13 grok13 statement13 Logstash13 offers13 an13 online13 tool13 at13 hcpsgrokdebugherokuappcom13

Translate13 If13 specific13 fields13 in13 the13 log13 record13 are13 coded13 (for13 example13 the13 Bluemix13 app13 name13 is13 oaen13 not13 sent13 in13 the13 syslog13 messages13 instead13 the13 GUID13 is13 sent13 which13 is13 not13 useful)13 then13 the13 filter13 secon13 translate13 plugin13 can13 be13 used13 to13 add13 new13 fields13 by13 holding13 the13 decoded13 versions13 of13 those13 values13

13 translate

field =gt data1

destination =gt data1

override =gt true

Page of 38 71 Page of 38 71

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

dc50f6e4-c052-436a-bf27-95ba392c4228 monverifyservicemanagement

0c677e82-5bfc-4e38-9155-e416249e3ee6 ms-catalog-postpyretic-pomiculture

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

However13 the13 translate13 plugin13 is13 not13 included13 with13 the13 Logstash13 installaon13 In13 the13 next13 secon13 I13 will13 13 describe13 how13 to13 add13 new13 plugins13 to13 the13 Logstash13 instance13

Add13 custom13 plugins13 to13 Logstash13 As13 an13 open13 source13 tool13 Logstash13 encourages13 users13 to13 develop13 content13 for13 all13 users13 to13 share13 Some13 of13 the13 plugins13 that13 were13 created13 by13 our13 community13 include13 a13 translaon13 plugin13 an13 output13 Slack13 plugin13 and13 an13 output13 syslog13 plugin13 Letrsquos13 look13 at13 how13 to13 add13 these13 plugins13 to13 your13 environment13

Filter13 translate13 plugin13 To13 add13 the13 translate13 plugin13

1 Download13 the13 plugin13

2 Copy13 the13 plugin13 file13 logstash-shy‐filter-shy‐translate-shy‐214gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-filter-translate 214

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Output13 Slack13 plugin13 To13 add13 the13 Slack13 plugin13

Page of 39 71 Page of 39 71

1 Download13 the13 plugin13 13

2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐slack-shy‐014gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-output-slack 014

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Output13 syslog13 plugin13 To13 add13 the13 syslog13 plugin13

1 Download13 the13 plugin13 13

2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐syslog-shy‐214gem13 to13 the13 ELK13 Virtual13 Server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-output-syslog 214

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Integrate13 Logstash13 with13 Slack13 To13 send13 generated13 alerts13 to13 Slack13 complete13 these13 steps13

1 Create13 a13 Slack13 integraon13 13

2 Use13 the13 Slack13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 Slack13 integraon13

113 Login13 to13 Slack13 and13 go13 to13 the13 Apps13 page13 13

Page of 40 71 Page of 40 71

13

313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13

13

A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13

13

Page of 42 71 Page of 42 71

The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13

13

The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13

13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

slack

url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5

channel =gt cloud-operations-mon

username =gt nick_cawood

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13

Page of 43 71 Page of 43 71

Page of 44 71 Page of 44 71

In13 Slack13 the13 alerts13 look13 like13 this13

13

Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13

1 Create13 a13 PagerDuty13 integraon13 13

2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13

1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13

13

2 Click13 Add13 New13 Service13

13

313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13

Page of 45 71 Page of 45 71

13

413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13

513 Click13 Add13 Service13

13

The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13

Page of 46 71 Page of 46 71

13

The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt be7eb1ea9c224018923e818d3e891d0b

incident_key =gt logstashhosttype

Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13

Page of 47 71 Page of 47 71

13

Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13

1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13

2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13

3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13

4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13

13 13 http

url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo

http_method =gt post

format =gt json

For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13

13 13 syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

Page of 48 71 Page of 48 71

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13

13

The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13

if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2

This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13

Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13

Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13

Logstash13 severity Shown13 in13 Omnibus13 as

alert Indeterminate

crical Warning

debug 7

emergency Clear

error Minor

informaonal 6

noce Crical

warning Major

Page of 49 71 Page of 49 71

13

Page of 50 71 Page of 50 71

Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13

input

tcp

port =gt 5000

type =gt syslog

filter

mutate

add_field =gt format =gt cf

grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

translate

field =gt data1

destination =gt data1

override =gt true

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

=== Check Alarm Criteria ===

Standard Key Words

if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

=== Alarm Exclusions ===

if [message] =~ (i)sometexthere

mutate remove_tag =gt servicename_alarm

Throttle alarms so we do not hit PD rate limits

Page of 52 71 Page of 52 71

if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 5

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle

after_count =gt 5

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 5

period =gt 1200

key =gt servicename_alarm_warning

add_tag =gt servicename_throttled

Page of 53 71 Page of 53 71

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

output

elasticsearch

hosts =gt localhost

stdout codec =gt rubydebug

if servicename_alarm_err in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

Page of 54 71 Page of 54 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 55 71 Page of 55 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

Page of 56 71 Page of 56 71

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert err message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_error in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

Page of 57 71 Page of 57 71

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 58 71 Page of 58 71

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 59 71 Page of 59 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert error message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

Page of 60 71 Page of 60 71

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

Page of 61 71 Page of 61 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 62 71 Page of 62 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt x

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert fatal message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

Page of 63 71 Page of 63 71

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

Page of 64 71 Page of 64 71

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 65 71 Page of 65 71

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt heavy_exclamation_mark

Page of 66 71 Page of 66 71

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert WARNING message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 67 71 Page of 67 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

Page of 68 71 Page of 68 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 69 71 Page of 69 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

Page of 70 71 Page of 70 71

Page of 71 71 Page of 71 71

Page 38: How to integrate the Elasticsearch Logstash Kibana (ELK ... · How to integrate the Elasticsearch Logstash Kibana (ELK) log analytics stack into IBM Bluemix Nick Cawood IBM Cloud

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

dc50f6e4-c052-436a-bf27-95ba392c4228 monverifyservicemanagement

0c677e82-5bfc-4e38-9155-e416249e3ee6 ms-catalog-postpyretic-pomiculture

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

However13 the13 translate13 plugin13 is13 not13 included13 with13 the13 Logstash13 installaon13 In13 the13 next13 secon13 I13 will13 13 describe13 how13 to13 add13 new13 plugins13 to13 the13 Logstash13 instance13

Add13 custom13 plugins13 to13 Logstash13 As13 an13 open13 source13 tool13 Logstash13 encourages13 users13 to13 develop13 content13 for13 all13 users13 to13 share13 Some13 of13 the13 plugins13 that13 were13 created13 by13 our13 community13 include13 a13 translaon13 plugin13 an13 output13 Slack13 plugin13 and13 an13 output13 syslog13 plugin13 Letrsquos13 look13 at13 how13 to13 add13 these13 plugins13 to13 your13 environment13

Filter13 translate13 plugin13 To13 add13 the13 translate13 plugin13

1 Download13 the13 plugin13

2 Copy13 the13 plugin13 file13 logstash-shy‐filter-shy‐translate-shy‐214gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-filter-translate 214

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Output13 Slack13 plugin13 To13 add13 the13 Slack13 plugin13

Page of 39 71 Page of 39 71

1 Download13 the13 plugin13 13

2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐slack-shy‐014gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-output-slack 014

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Output13 syslog13 plugin13 To13 add13 the13 syslog13 plugin13

1 Download13 the13 plugin13 13

2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐syslog-shy‐214gem13 to13 the13 ELK13 Virtual13 Server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-output-syslog 214

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Integrate13 Logstash13 with13 Slack13 To13 send13 generated13 alerts13 to13 Slack13 complete13 these13 steps13

1 Create13 a13 Slack13 integraon13 13

2 Use13 the13 Slack13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 Slack13 integraon13

113 Login13 to13 Slack13 and13 go13 to13 the13 Apps13 page13 13

Page of 40 71 Page of 40 71

13

313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13

13

A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13

13

Page of 42 71 Page of 42 71

The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13

13

The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13

13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

slack

url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5

channel =gt cloud-operations-mon

username =gt nick_cawood

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13

Page of 43 71 Page of 43 71

Page of 44 71 Page of 44 71

In13 Slack13 the13 alerts13 look13 like13 this13

13

Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13

1 Create13 a13 PagerDuty13 integraon13 13

2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13

1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13

13

2 Click13 Add13 New13 Service13

13

313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13

Page of 45 71 Page of 45 71

13

413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13

513 Click13 Add13 Service13

13

The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13

Page of 46 71 Page of 46 71

13

The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt be7eb1ea9c224018923e818d3e891d0b

incident_key =gt logstashhosttype

Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13

Page of 47 71 Page of 47 71

13

Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13

1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13

2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13

3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13

4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13

13 13 http

url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo

http_method =gt post

format =gt json

For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13

13 13 syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

Page of 48 71 Page of 48 71

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13

13

The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13

if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2

This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13

Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13

Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13

Logstash13 severity Shown13 in13 Omnibus13 as

alert Indeterminate

crical Warning

debug 7

emergency Clear

error Minor

informaonal 6

noce Crical

warning Major

Page of 49 71 Page of 49 71

13

Page of 50 71 Page of 50 71

Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13

input

tcp

port =gt 5000

type =gt syslog

filter

mutate

add_field =gt format =gt cf

grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

translate

field =gt data1

destination =gt data1

override =gt true

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

=== Check Alarm Criteria ===

Standard Key Words

if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

=== Alarm Exclusions ===

if [message] =~ (i)sometexthere

mutate remove_tag =gt servicename_alarm

Throttle alarms so we do not hit PD rate limits

Page of 52 71 Page of 52 71

if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 5

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle

after_count =gt 5

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 5

period =gt 1200

key =gt servicename_alarm_warning

add_tag =gt servicename_throttled

Page of 53 71 Page of 53 71

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

output

elasticsearch

hosts =gt localhost

stdout codec =gt rubydebug

if servicename_alarm_err in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

Page of 54 71 Page of 54 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 55 71 Page of 55 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

Page of 56 71 Page of 56 71

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert err message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_error in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

Page of 57 71 Page of 57 71

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 58 71 Page of 58 71

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 59 71 Page of 59 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert error message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

Page of 60 71 Page of 60 71

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

Page of 61 71 Page of 61 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 62 71 Page of 62 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt x

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert fatal message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

Page of 63 71 Page of 63 71

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

Page of 64 71 Page of 64 71

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 65 71 Page of 65 71

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt heavy_exclamation_mark

Page of 66 71 Page of 66 71

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert WARNING message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 67 71 Page of 67 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

Page of 68 71 Page of 68 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 69 71 Page of 69 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

Page of 70 71 Page of 70 71

Page of 71 71 Page of 71 71

Page 39: How to integrate the Elasticsearch Logstash Kibana (ELK ... · How to integrate the Elasticsearch Logstash Kibana (ELK) log analytics stack into IBM Bluemix Nick Cawood IBM Cloud

1 Download13 the13 plugin13 13

2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐slack-shy‐014gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-output-slack 014

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Output13 syslog13 plugin13 To13 add13 the13 syslog13 plugin13

1 Download13 the13 plugin13 13

2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐syslog-shy‐214gem13 to13 the13 ELK13 Virtual13 Server13 (see13 previous13 secon13 in13 this13 document)13

3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13

gem logstash-output-syslog 214

4 Run13 the13 command13

[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify

The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13

Integrate13 Logstash13 with13 Slack13 To13 send13 generated13 alerts13 to13 Slack13 complete13 these13 steps13

1 Create13 a13 Slack13 integraon13 13

2 Use13 the13 Slack13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 Slack13 integraon13

113 Login13 to13 Slack13 and13 go13 to13 the13 Apps13 page13 13

Page of 40 71 Page of 40 71

13

313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13

13

A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13

13

Page of 42 71 Page of 42 71

The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13

13

The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13

13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

slack

url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5

channel =gt cloud-operations-mon

username =gt nick_cawood

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13

Page of 43 71 Page of 43 71

Page of 44 71 Page of 44 71

In13 Slack13 the13 alerts13 look13 like13 this13

13

Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13

1 Create13 a13 PagerDuty13 integraon13 13

2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13

1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13

13

2 Click13 Add13 New13 Service13

13

313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13

Page of 45 71 Page of 45 71

13

413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13

513 Click13 Add13 Service13

13

The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13

Page of 46 71 Page of 46 71

13

The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt be7eb1ea9c224018923e818d3e891d0b

incident_key =gt logstashhosttype

Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13

Page of 47 71 Page of 47 71

13

Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13

1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13

2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13

3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13

4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13

13 13 http

url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo

http_method =gt post

format =gt json

For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13

13 13 syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

Page of 48 71 Page of 48 71

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13

13

The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13

if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2

This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13

Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13

Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13

Logstash13 severity Shown13 in13 Omnibus13 as

alert Indeterminate

crical Warning

debug 7

emergency Clear

error Minor

informaonal 6

noce Crical

warning Major

Page of 49 71 Page of 49 71

13

Page of 50 71 Page of 50 71

Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13

input

tcp

port =gt 5000

type =gt syslog

filter

mutate

add_field =gt format =gt cf

grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

translate

field =gt data1

destination =gt data1

override =gt true

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

=== Check Alarm Criteria ===

Standard Key Words

if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

=== Alarm Exclusions ===

if [message] =~ (i)sometexthere

mutate remove_tag =gt servicename_alarm

Throttle alarms so we do not hit PD rate limits

Page of 52 71 Page of 52 71

if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 5

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle

after_count =gt 5

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 5

period =gt 1200

key =gt servicename_alarm_warning

add_tag =gt servicename_throttled

Page of 53 71 Page of 53 71

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

output

elasticsearch

hosts =gt localhost

stdout codec =gt rubydebug

if servicename_alarm_err in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

Page of 54 71 Page of 54 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 55 71 Page of 55 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

Page of 56 71 Page of 56 71

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert err message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_error in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

Page of 57 71 Page of 57 71

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 58 71 Page of 58 71

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 59 71 Page of 59 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert error message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

Page of 60 71 Page of 60 71

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

Page of 61 71 Page of 61 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 62 71 Page of 62 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt x

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert fatal message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

Page of 63 71 Page of 63 71

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

Page of 64 71 Page of 64 71

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 65 71 Page of 65 71

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt heavy_exclamation_mark

Page of 66 71 Page of 66 71

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert WARNING message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 67 71 Page of 67 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

Page of 68 71 Page of 68 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 69 71 Page of 69 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

Page of 70 71 Page of 70 71

Page of 71 71 Page of 71 71

Page 40: How to integrate the Elasticsearch Logstash Kibana (ELK ... · How to integrate the Elasticsearch Logstash Kibana (ELK) log analytics stack into IBM Bluemix Nick Cawood IBM Cloud

13

313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13

13

A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13

13

Page of 42 71 Page of 42 71

The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13

13

The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13

13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

slack

url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5

channel =gt cloud-operations-mon

username =gt nick_cawood

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13

Page of 43 71 Page of 43 71

Page of 44 71 Page of 44 71

In13 Slack13 the13 alerts13 look13 like13 this13

13

Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13

1 Create13 a13 PagerDuty13 integraon13 13

2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13

1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13

13

2 Click13 Add13 New13 Service13

13

313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13

Page of 45 71 Page of 45 71

13

413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13

513 Click13 Add13 Service13

13

The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13

Page of 46 71 Page of 46 71

13

The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt be7eb1ea9c224018923e818d3e891d0b

incident_key =gt logstashhosttype

Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13

Page of 47 71 Page of 47 71

13

Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13

1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13

2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13

3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13

4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13

13 13 http

url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo

http_method =gt post

format =gt json

For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13

13 13 syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

Page of 48 71 Page of 48 71

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13

13

The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13

if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2

This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13

Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13

Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13

Logstash13 severity Shown13 in13 Omnibus13 as

alert Indeterminate

crical Warning

debug 7

emergency Clear

error Minor

informaonal 6

noce Crical

warning Major

Page of 49 71 Page of 49 71

13

Page of 50 71 Page of 50 71

Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13

input

tcp

port =gt 5000

type =gt syslog

filter

mutate

add_field =gt format =gt cf

grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

translate

field =gt data1

destination =gt data1

override =gt true

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

=== Check Alarm Criteria ===

Standard Key Words

if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

=== Alarm Exclusions ===

if [message] =~ (i)sometexthere

mutate remove_tag =gt servicename_alarm

Throttle alarms so we do not hit PD rate limits

Page of 52 71 Page of 52 71

if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 5

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle

after_count =gt 5

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 5

period =gt 1200

key =gt servicename_alarm_warning

add_tag =gt servicename_throttled

Page of 53 71 Page of 53 71

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

output

elasticsearch

hosts =gt localhost

stdout codec =gt rubydebug

if servicename_alarm_err in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

Page of 54 71 Page of 54 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 55 71 Page of 55 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

Page of 56 71 Page of 56 71

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert err message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_error in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

Page of 57 71 Page of 57 71

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 58 71 Page of 58 71

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 59 71 Page of 59 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert error message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

Page of 60 71 Page of 60 71

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

Page of 61 71 Page of 61 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 62 71 Page of 62 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt x

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert fatal message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

Page of 63 71 Page of 63 71

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

Page of 64 71 Page of 64 71

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 65 71 Page of 65 71

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt heavy_exclamation_mark

Page of 66 71 Page of 66 71

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert WARNING message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 67 71 Page of 67 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

Page of 68 71 Page of 68 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 69 71 Page of 69 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

Page of 70 71 Page of 70 71

Page of 71 71 Page of 71 71

Page 41: How to integrate the Elasticsearch Logstash Kibana (ELK ... · How to integrate the Elasticsearch Logstash Kibana (ELK) log analytics stack into IBM Bluemix Nick Cawood IBM Cloud

The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13

13

The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13

13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

slack

url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5

channel =gt cloud-operations-mon

username =gt nick_cawood

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13

Page of 43 71 Page of 43 71

Page of 44 71 Page of 44 71

In13 Slack13 the13 alerts13 look13 like13 this13

13

Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13

1 Create13 a13 PagerDuty13 integraon13 13

2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13

1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13

13

2 Click13 Add13 New13 Service13

13

313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13

Page of 45 71 Page of 45 71

13

413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13

513 Click13 Add13 Service13

13

The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13

Page of 46 71 Page of 46 71

13

The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt be7eb1ea9c224018923e818d3e891d0b

incident_key =gt logstashhosttype

Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13

Page of 47 71 Page of 47 71

13

Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13

1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13

2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13

3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13

4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13

13 13 http

url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo

http_method =gt post

format =gt json

For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13

13 13 syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

Page of 48 71 Page of 48 71

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13

13

The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13

if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2

This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13

Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13

Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13

Logstash13 severity Shown13 in13 Omnibus13 as

alert Indeterminate

crical Warning

debug 7

emergency Clear

error Minor

informaonal 6

noce Crical

warning Major

Page of 49 71 Page of 49 71

13

Page of 50 71 Page of 50 71

Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13

input

tcp

port =gt 5000

type =gt syslog

filter

mutate

add_field =gt format =gt cf

grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

translate

field =gt data1

destination =gt data1

override =gt true

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

=== Check Alarm Criteria ===

Standard Key Words

if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

=== Alarm Exclusions ===

if [message] =~ (i)sometexthere

mutate remove_tag =gt servicename_alarm

Throttle alarms so we do not hit PD rate limits

Page of 52 71 Page of 52 71

if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 5

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle

after_count =gt 5

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 5

period =gt 1200

key =gt servicename_alarm_warning

add_tag =gt servicename_throttled

Page of 53 71 Page of 53 71

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

output

elasticsearch

hosts =gt localhost

stdout codec =gt rubydebug

if servicename_alarm_err in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

Page of 54 71 Page of 54 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 55 71 Page of 55 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

Page of 56 71 Page of 56 71

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert err message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_error in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

Page of 57 71 Page of 57 71

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 58 71 Page of 58 71

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 59 71 Page of 59 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert error message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

Page of 60 71 Page of 60 71

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

Page of 61 71 Page of 61 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 62 71 Page of 62 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt x

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert fatal message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

Page of 63 71 Page of 63 71

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

Page of 64 71 Page of 64 71

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 65 71 Page of 65 71

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt heavy_exclamation_mark

Page of 66 71 Page of 66 71

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert WARNING message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 67 71 Page of 67 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

Page of 68 71 Page of 68 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 69 71 Page of 69 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

Page of 70 71 Page of 70 71

Page of 71 71 Page of 71 71

Page 42: How to integrate the Elasticsearch Logstash Kibana (ELK ... · How to integrate the Elasticsearch Logstash Kibana (ELK) log analytics stack into IBM Bluemix Nick Cawood IBM Cloud

Page of 44 71 Page of 44 71

In13 Slack13 the13 alerts13 look13 like13 this13

13

Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13

1 Create13 a13 PagerDuty13 integraon13 13

2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13

1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13

13

2 Click13 Add13 New13 Service13

13

313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13

Page of 45 71 Page of 45 71

13

413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13

513 Click13 Add13 Service13

13

The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13

Page of 46 71 Page of 46 71

13

The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt be7eb1ea9c224018923e818d3e891d0b

incident_key =gt logstashhosttype

Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13

Page of 47 71 Page of 47 71

13

Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13

1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13

2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13

3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13

4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13

13 13 http

url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo

http_method =gt post

format =gt json

For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13

13 13 syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

Page of 48 71 Page of 48 71

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13

13

The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13

if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2

This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13

Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13

Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13

Logstash13 severity Shown13 in13 Omnibus13 as

alert Indeterminate

crical Warning

debug 7

emergency Clear

error Minor

informaonal 6

noce Crical

warning Major

Page of 49 71 Page of 49 71

13

Page of 50 71 Page of 50 71

Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13

input

tcp

port =gt 5000

type =gt syslog

filter

mutate

add_field =gt format =gt cf

grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

translate

field =gt data1

destination =gt data1

override =gt true

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

=== Check Alarm Criteria ===

Standard Key Words

if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

=== Alarm Exclusions ===

if [message] =~ (i)sometexthere

mutate remove_tag =gt servicename_alarm

Throttle alarms so we do not hit PD rate limits

Page of 52 71 Page of 52 71

if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 5

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle

after_count =gt 5

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 5

period =gt 1200

key =gt servicename_alarm_warning

add_tag =gt servicename_throttled

Page of 53 71 Page of 53 71

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

output

elasticsearch

hosts =gt localhost

stdout codec =gt rubydebug

if servicename_alarm_err in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

Page of 54 71 Page of 54 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 55 71 Page of 55 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

Page of 56 71 Page of 56 71

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert err message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_error in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

Page of 57 71 Page of 57 71

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 58 71 Page of 58 71

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 59 71 Page of 59 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert error message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

Page of 60 71 Page of 60 71

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

Page of 61 71 Page of 61 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 62 71 Page of 62 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt x

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert fatal message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

Page of 63 71 Page of 63 71

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

Page of 64 71 Page of 64 71

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 65 71 Page of 65 71

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt heavy_exclamation_mark

Page of 66 71 Page of 66 71

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert WARNING message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 67 71 Page of 67 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

Page of 68 71 Page of 68 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 69 71 Page of 69 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

Page of 70 71 Page of 70 71

Page of 71 71 Page of 71 71

Page 43: How to integrate the Elasticsearch Logstash Kibana (ELK ... · How to integrate the Elasticsearch Logstash Kibana (ELK) log analytics stack into IBM Bluemix Nick Cawood IBM Cloud

In13 Slack13 the13 alerts13 look13 like13 this13

13

Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13

1 Create13 a13 PagerDuty13 integraon13 13

2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13

To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13

1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13

13

2 Click13 Add13 New13 Service13

13

313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13

Page of 45 71 Page of 45 71

13

413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13

513 Click13 Add13 Service13

13

The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13

Page of 46 71 Page of 46 71

13

The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt be7eb1ea9c224018923e818d3e891d0b

incident_key =gt logstashhosttype

Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13

Page of 47 71 Page of 47 71

13

Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13

1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13

2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13

3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13

4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13

13 13 http

url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo

http_method =gt post

format =gt json

For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13

13 13 syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

Page of 48 71 Page of 48 71

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13

13

The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13

if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2

This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13

Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13

Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13

Logstash13 severity Shown13 in13 Omnibus13 as

alert Indeterminate

crical Warning

debug 7

emergency Clear

error Minor

informaonal 6

noce Crical

warning Major

Page of 49 71 Page of 49 71

13

Page of 50 71 Page of 50 71

Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13

input

tcp

port =gt 5000

type =gt syslog

filter

mutate

add_field =gt format =gt cf

grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

translate

field =gt data1

destination =gt data1

override =gt true

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

=== Check Alarm Criteria ===

Standard Key Words

if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

=== Alarm Exclusions ===

if [message] =~ (i)sometexthere

mutate remove_tag =gt servicename_alarm

Throttle alarms so we do not hit PD rate limits

Page of 52 71 Page of 52 71

if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 5

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle

after_count =gt 5

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 5

period =gt 1200

key =gt servicename_alarm_warning

add_tag =gt servicename_throttled

Page of 53 71 Page of 53 71

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

output

elasticsearch

hosts =gt localhost

stdout codec =gt rubydebug

if servicename_alarm_err in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

Page of 54 71 Page of 54 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 55 71 Page of 55 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

Page of 56 71 Page of 56 71

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert err message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_error in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

Page of 57 71 Page of 57 71

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 58 71 Page of 58 71

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 59 71 Page of 59 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert error message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

Page of 60 71 Page of 60 71

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

Page of 61 71 Page of 61 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 62 71 Page of 62 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt x

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert fatal message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

Page of 63 71 Page of 63 71

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

Page of 64 71 Page of 64 71

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 65 71 Page of 65 71

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt heavy_exclamation_mark

Page of 66 71 Page of 66 71

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert WARNING message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 67 71 Page of 67 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

Page of 68 71 Page of 68 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 69 71 Page of 69 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

Page of 70 71 Page of 70 71

Page of 71 71 Page of 71 71

Page 44: How to integrate the Elasticsearch Logstash Kibana (ELK ... · How to integrate the Elasticsearch Logstash Kibana (ELK) log analytics stack into IBM Bluemix Nick Cawood IBM Cloud

13

413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13

513 Click13 Add13 Service13

13

The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13

Page of 46 71 Page of 46 71

13

The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt be7eb1ea9c224018923e818d3e891d0b

incident_key =gt logstashhosttype

Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13

Page of 47 71 Page of 47 71

13

Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13

1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13

2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13

3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13

4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13

13 13 http

url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo

http_method =gt post

format =gt json

For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13

13 13 syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

Page of 48 71 Page of 48 71

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13

13

The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13

if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2

This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13

Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13

Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13

Logstash13 severity Shown13 in13 Omnibus13 as

alert Indeterminate

crical Warning

debug 7

emergency Clear

error Minor

informaonal 6

noce Crical

warning Major

Page of 49 71 Page of 49 71

13

Page of 50 71 Page of 50 71

Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13

input

tcp

port =gt 5000

type =gt syslog

filter

mutate

add_field =gt format =gt cf

grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

translate

field =gt data1

destination =gt data1

override =gt true

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

=== Check Alarm Criteria ===

Standard Key Words

if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

=== Alarm Exclusions ===

if [message] =~ (i)sometexthere

mutate remove_tag =gt servicename_alarm

Throttle alarms so we do not hit PD rate limits

Page of 52 71 Page of 52 71

if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 5

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle

after_count =gt 5

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 5

period =gt 1200

key =gt servicename_alarm_warning

add_tag =gt servicename_throttled

Page of 53 71 Page of 53 71

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

output

elasticsearch

hosts =gt localhost

stdout codec =gt rubydebug

if servicename_alarm_err in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

Page of 54 71 Page of 54 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 55 71 Page of 55 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

Page of 56 71 Page of 56 71

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert err message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_error in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

Page of 57 71 Page of 57 71

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 58 71 Page of 58 71

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 59 71 Page of 59 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert error message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

Page of 60 71 Page of 60 71

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

Page of 61 71 Page of 61 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 62 71 Page of 62 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt x

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert fatal message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

Page of 63 71 Page of 63 71

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

Page of 64 71 Page of 64 71

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 65 71 Page of 65 71

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt heavy_exclamation_mark

Page of 66 71 Page of 66 71

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert WARNING message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 67 71 Page of 67 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

Page of 68 71 Page of 68 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 69 71 Page of 69 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

Page of 70 71 Page of 70 71

Page of 71 71 Page of 71 71

Page 45: How to integrate the Elasticsearch Logstash Kibana (ELK ... · How to integrate the Elasticsearch Logstash Kibana (ELK) log analytics stack into IBM Bluemix Nick Cawood IBM Cloud

13

The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt be7eb1ea9c224018923e818d3e891d0b

incident_key =gt logstashhosttype

Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13

Page of 47 71 Page of 47 71

13

Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13

1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13

2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13

3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13

4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13

13 13 http

url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo

http_method =gt post

format =gt json

For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13

13 13 syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

Page of 48 71 Page of 48 71

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13

13

The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13

if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2

This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13

Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13

Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13

Logstash13 severity Shown13 in13 Omnibus13 as

alert Indeterminate

crical Warning

debug 7

emergency Clear

error Minor

informaonal 6

noce Crical

warning Major

Page of 49 71 Page of 49 71

13

Page of 50 71 Page of 50 71

Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13

input

tcp

port =gt 5000

type =gt syslog

filter

mutate

add_field =gt format =gt cf

grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

translate

field =gt data1

destination =gt data1

override =gt true

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

=== Check Alarm Criteria ===

Standard Key Words

if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

=== Alarm Exclusions ===

if [message] =~ (i)sometexthere

mutate remove_tag =gt servicename_alarm

Throttle alarms so we do not hit PD rate limits

Page of 52 71 Page of 52 71

if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 5

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle

after_count =gt 5

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 5

period =gt 1200

key =gt servicename_alarm_warning

add_tag =gt servicename_throttled

Page of 53 71 Page of 53 71

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

output

elasticsearch

hosts =gt localhost

stdout codec =gt rubydebug

if servicename_alarm_err in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

Page of 54 71 Page of 54 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 55 71 Page of 55 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

Page of 56 71 Page of 56 71

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert err message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_error in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

Page of 57 71 Page of 57 71

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 58 71 Page of 58 71

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 59 71 Page of 59 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert error message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

Page of 60 71 Page of 60 71

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

Page of 61 71 Page of 61 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 62 71 Page of 62 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt x

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert fatal message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

Page of 63 71 Page of 63 71

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

Page of 64 71 Page of 64 71

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 65 71 Page of 65 71

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt heavy_exclamation_mark

Page of 66 71 Page of 66 71

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert WARNING message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 67 71 Page of 67 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

Page of 68 71 Page of 68 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 69 71 Page of 69 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

Page of 70 71 Page of 70 71

Page of 71 71 Page of 71 71

Page 46: How to integrate the Elasticsearch Logstash Kibana (ELK ... · How to integrate the Elasticsearch Logstash Kibana (ELK) log analytics stack into IBM Bluemix Nick Cawood IBM Cloud

13

Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13

1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13

2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13

3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13

4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13

13 13 http

url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo

http_method =gt post

format =gt json

For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13

13 13 syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

Page of 48 71 Page of 48 71

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13

13

The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13

if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2

This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13

Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13

Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13

Logstash13 severity Shown13 in13 Omnibus13 as

alert Indeterminate

crical Warning

debug 7

emergency Clear

error Minor

informaonal 6

noce Crical

warning Major

Page of 49 71 Page of 49 71

13

Page of 50 71 Page of 50 71

Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13

input

tcp

port =gt 5000

type =gt syslog

filter

mutate

add_field =gt format =gt cf

grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

translate

field =gt data1

destination =gt data1

override =gt true

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

=== Check Alarm Criteria ===

Standard Key Words

if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

=== Alarm Exclusions ===

if [message] =~ (i)sometexthere

mutate remove_tag =gt servicename_alarm

Throttle alarms so we do not hit PD rate limits

Page of 52 71 Page of 52 71

if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 5

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle

after_count =gt 5

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 5

period =gt 1200

key =gt servicename_alarm_warning

add_tag =gt servicename_throttled

Page of 53 71 Page of 53 71

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

output

elasticsearch

hosts =gt localhost

stdout codec =gt rubydebug

if servicename_alarm_err in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

Page of 54 71 Page of 54 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 55 71 Page of 55 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

Page of 56 71 Page of 56 71

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert err message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_error in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

Page of 57 71 Page of 57 71

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 58 71 Page of 58 71

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 59 71 Page of 59 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert error message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

Page of 60 71 Page of 60 71

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

Page of 61 71 Page of 61 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 62 71 Page of 62 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt x

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert fatal message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

Page of 63 71 Page of 63 71

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

Page of 64 71 Page of 64 71

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 65 71 Page of 65 71

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt heavy_exclamation_mark

Page of 66 71 Page of 66 71

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert WARNING message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 67 71 Page of 67 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

Page of 68 71 Page of 68 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 69 71 Page of 69 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

Page of 70 71 Page of 70 71

Page of 71 71 Page of 71 71

Page 47: How to integrate the Elasticsearch Logstash Kibana (ELK ... · How to integrate the Elasticsearch Logstash Kibana (ELK) log analytics stack into IBM Bluemix Nick Cawood IBM Cloud

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13

13

The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13

if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2

This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13

Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13

Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13

Logstash13 severity Shown13 in13 Omnibus13 as

alert Indeterminate

crical Warning

debug 7

emergency Clear

error Minor

informaonal 6

noce Crical

warning Major

Page of 49 71 Page of 49 71

13

Page of 50 71 Page of 50 71

Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13

input

tcp

port =gt 5000

type =gt syslog

filter

mutate

add_field =gt format =gt cf

grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

translate

field =gt data1

destination =gt data1

override =gt true

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

=== Check Alarm Criteria ===

Standard Key Words

if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

=== Alarm Exclusions ===

if [message] =~ (i)sometexthere

mutate remove_tag =gt servicename_alarm

Throttle alarms so we do not hit PD rate limits

Page of 52 71 Page of 52 71

if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 5

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle

after_count =gt 5

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 5

period =gt 1200

key =gt servicename_alarm_warning

add_tag =gt servicename_throttled

Page of 53 71 Page of 53 71

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

output

elasticsearch

hosts =gt localhost

stdout codec =gt rubydebug

if servicename_alarm_err in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

Page of 54 71 Page of 54 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 55 71 Page of 55 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

Page of 56 71 Page of 56 71

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert err message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_error in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

Page of 57 71 Page of 57 71

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 58 71 Page of 58 71

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 59 71 Page of 59 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert error message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

Page of 60 71 Page of 60 71

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

Page of 61 71 Page of 61 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 62 71 Page of 62 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt x

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert fatal message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

Page of 63 71 Page of 63 71

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

Page of 64 71 Page of 64 71

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 65 71 Page of 65 71

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt heavy_exclamation_mark

Page of 66 71 Page of 66 71

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert WARNING message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 67 71 Page of 67 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

Page of 68 71 Page of 68 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 69 71 Page of 69 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

Page of 70 71 Page of 70 71

Page of 71 71 Page of 71 71

Page 48: How to integrate the Elasticsearch Logstash Kibana (ELK ... · How to integrate the Elasticsearch Logstash Kibana (ELK) log analytics stack into IBM Bluemix Nick Cawood IBM Cloud

13

Page of 50 71 Page of 50 71

Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13

input

tcp

port =gt 5000

type =gt syslog

filter

mutate

add_field =gt format =gt cf

grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

translate

field =gt data1

destination =gt data1

override =gt true

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

=== Check Alarm Criteria ===

Standard Key Words

if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

=== Alarm Exclusions ===

if [message] =~ (i)sometexthere

mutate remove_tag =gt servicename_alarm

Throttle alarms so we do not hit PD rate limits

Page of 52 71 Page of 52 71

if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 5

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle

after_count =gt 5

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 5

period =gt 1200

key =gt servicename_alarm_warning

add_tag =gt servicename_throttled

Page of 53 71 Page of 53 71

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

output

elasticsearch

hosts =gt localhost

stdout codec =gt rubydebug

if servicename_alarm_err in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

Page of 54 71 Page of 54 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 55 71 Page of 55 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

Page of 56 71 Page of 56 71

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert err message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_error in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

Page of 57 71 Page of 57 71

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 58 71 Page of 58 71

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 59 71 Page of 59 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert error message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

Page of 60 71 Page of 60 71

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

Page of 61 71 Page of 61 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 62 71 Page of 62 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt x

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert fatal message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

Page of 63 71 Page of 63 71

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

Page of 64 71 Page of 64 71

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 65 71 Page of 65 71

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt heavy_exclamation_mark

Page of 66 71 Page of 66 71

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert WARNING message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 67 71 Page of 67 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

Page of 68 71 Page of 68 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 69 71 Page of 69 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

Page of 70 71 Page of 70 71

Page of 71 71 Page of 71 71

Page 49: How to integrate the Elasticsearch Logstash Kibana (ELK ... · How to integrate the Elasticsearch Logstash Kibana (ELK) log analytics stack into IBM Bluemix Nick Cawood IBM Cloud

Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13

input

tcp

port =gt 5000

type =gt syslog

filter

mutate

add_field =gt format =gt cf

grok

match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend

add_tag =gt [grokked]

end grok

translate

field =gt data1

destination =gt data1

override =gt true

dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

=== Check Alarm Criteria ===

Standard Key Words

if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

=== Alarm Exclusions ===

if [message] =~ (i)sometexthere

mutate remove_tag =gt servicename_alarm

Throttle alarms so we do not hit PD rate limits

Page of 52 71 Page of 52 71

if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 5

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle

after_count =gt 5

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 5

period =gt 1200

key =gt servicename_alarm_warning

add_tag =gt servicename_throttled

Page of 53 71 Page of 53 71

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

output

elasticsearch

hosts =gt localhost

stdout codec =gt rubydebug

if servicename_alarm_err in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

Page of 54 71 Page of 54 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 55 71 Page of 55 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

Page of 56 71 Page of 56 71

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert err message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_error in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

Page of 57 71 Page of 57 71

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 58 71 Page of 58 71

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 59 71 Page of 59 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert error message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

Page of 60 71 Page of 60 71

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

Page of 61 71 Page of 61 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 62 71 Page of 62 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt x

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert fatal message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

Page of 63 71 Page of 63 71

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

Page of 64 71 Page of 64 71

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 65 71 Page of 65 71

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt heavy_exclamation_mark

Page of 66 71 Page of 66 71

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert WARNING message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 67 71 Page of 67 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

Page of 68 71 Page of 68 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 69 71 Page of 69 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

Page of 70 71 Page of 70 71

Page of 71 71 Page of 71 71

Page 50: How to integrate the Elasticsearch Logstash Kibana (ELK ... · How to integrate the Elasticsearch Logstash Kibana (ELK) log analytics stack into IBM Bluemix Nick Cawood IBM Cloud

f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615

059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639

9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]

add_tag =gt translated

=== Check Alarm Criteria ===

Standard Key Words

if [message] =~ (err)

mutate add_tag =gt servicename_alarm_err

if [message] =~ (Error)

mutate add_tag =gt servicename_alarm_error

if [message] =~ (fatal)

mutate add_tag =gt servicename_alarm_fatal

if [message] =~ (WARNING)

mutate add_tag =gt servicename_alarm_warning

if [message] =~ (STOPPED|Stopped|CRASHED)

mutate add_tag =gt servicename_alarm_stopped

=== Alarm Exclusions ===

if [message] =~ (i)sometexthere

mutate remove_tag =gt servicename_alarm

Throttle alarms so we do not hit PD rate limits

Page of 52 71 Page of 52 71

if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 5

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle

after_count =gt 5

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 5

period =gt 1200

key =gt servicename_alarm_warning

add_tag =gt servicename_throttled

Page of 53 71 Page of 53 71

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

output

elasticsearch

hosts =gt localhost

stdout codec =gt rubydebug

if servicename_alarm_err in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

Page of 54 71 Page of 54 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 55 71 Page of 55 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

Page of 56 71 Page of 56 71

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert err message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_error in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

Page of 57 71 Page of 57 71

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 58 71 Page of 58 71

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 59 71 Page of 59 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert error message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

Page of 60 71 Page of 60 71

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

Page of 61 71 Page of 61 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 62 71 Page of 62 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt x

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert fatal message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

Page of 63 71 Page of 63 71

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

Page of 64 71 Page of 64 71

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 65 71 Page of 65 71

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt heavy_exclamation_mark

Page of 66 71 Page of 66 71

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert WARNING message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 67 71 Page of 67 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

Page of 68 71 Page of 68 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 69 71 Page of 69 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

Page of 70 71 Page of 70 71

Page of 71 71 Page of 71 71

Page 51: How to integrate the Elasticsearch Logstash Kibana (ELK ... · How to integrate the Elasticsearch Logstash Kibana (ELK) log analytics stack into IBM Bluemix Nick Cawood IBM Cloud

if servicename_alarm_err in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_err

add_tag =gt servicename_throttled

if servicename_alarm_error in [tags]

throttle

after_count =gt 5

period =gt 600

key =gt servicename_alarm_error

add_tag =gt servicename_throttled

if servicename_alarm_fatal in [tags]

throttle

after_count =gt 5

period =gt 900

key =gt servicename_alarm_fatal

add_tag =gt servicename_throttled

if servicename_alarm_warning in [tags]

throttle

after_count =gt 5

period =gt 1200

key =gt servicename_alarm_warning

add_tag =gt servicename_throttled

Page of 53 71 Page of 53 71

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

output

elasticsearch

hosts =gt localhost

stdout codec =gt rubydebug

if servicename_alarm_err in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

Page of 54 71 Page of 54 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 55 71 Page of 55 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

Page of 56 71 Page of 56 71

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert err message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_error in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

Page of 57 71 Page of 57 71

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 58 71 Page of 58 71

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 59 71 Page of 59 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert error message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

Page of 60 71 Page of 60 71

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

Page of 61 71 Page of 61 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 62 71 Page of 62 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt x

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert fatal message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

Page of 63 71 Page of 63 71

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

Page of 64 71 Page of 64 71

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 65 71 Page of 65 71

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt heavy_exclamation_mark

Page of 66 71 Page of 66 71

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert WARNING message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 67 71 Page of 67 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

Page of 68 71 Page of 68 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 69 71 Page of 69 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

Page of 70 71 Page of 70 71

Page of 71 71 Page of 71 71

Page 52: How to integrate the Elasticsearch Logstash Kibana (ELK ... · How to integrate the Elasticsearch Logstash Kibana (ELK) log analytics stack into IBM Bluemix Nick Cawood IBM Cloud

if servicename_alarm_stopped in [tags]

throttle

after_count =gt 5

period =gt 300

key =gt servicename_alarm_stopped

add_tag =gt servicename_throttled

output

elasticsearch

hosts =gt localhost

stdout codec =gt rubydebug

if servicename_alarm_err in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

Page of 54 71 Page of 54 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 55 71 Page of 55 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

Page of 56 71 Page of 56 71

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert err message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_error in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

Page of 57 71 Page of 57 71

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 58 71 Page of 58 71

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 59 71 Page of 59 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert error message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

Page of 60 71 Page of 60 71

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

Page of 61 71 Page of 61 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 62 71 Page of 62 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt x

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert fatal message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

Page of 63 71 Page of 63 71

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

Page of 64 71 Page of 64 71

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 65 71 Page of 65 71

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt heavy_exclamation_mark

Page of 66 71 Page of 66 71

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert WARNING message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 67 71 Page of 67 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

Page of 68 71 Page of 68 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 69 71 Page of 69 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

Page of 70 71 Page of 70 71

Page of 71 71 Page of 71 71

Page 53: How to integrate the Elasticsearch Logstash Kibana (ELK ... · How to integrate the Elasticsearch Logstash Kibana (ELK) log analytics stack into IBM Bluemix Nick Cawood IBM Cloud

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 55 71 Page of 55 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

Page of 56 71 Page of 56 71

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert err message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_error in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

Page of 57 71 Page of 57 71

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 58 71 Page of 58 71

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 59 71 Page of 59 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert error message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

Page of 60 71 Page of 60 71

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

Page of 61 71 Page of 61 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 62 71 Page of 62 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt x

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert fatal message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

Page of 63 71 Page of 63 71

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

Page of 64 71 Page of 64 71

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 65 71 Page of 65 71

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt heavy_exclamation_mark

Page of 66 71 Page of 66 71

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert WARNING message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 67 71 Page of 67 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

Page of 68 71 Page of 68 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 69 71 Page of 69 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

Page of 70 71 Page of 70 71

Page of 71 71 Page of 71 71

Page 54: How to integrate the Elasticsearch Logstash Kibana (ELK ... · How to integrate the Elasticsearch Logstash Kibana (ELK) log analytics stack into IBM Bluemix Nick Cawood IBM Cloud

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

Page of 56 71 Page of 56 71

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert err message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_error in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

Page of 57 71 Page of 57 71

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 58 71 Page of 58 71

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 59 71 Page of 59 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert error message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

Page of 60 71 Page of 60 71

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

Page of 61 71 Page of 61 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 62 71 Page of 62 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt x

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert fatal message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

Page of 63 71 Page of 63 71

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

Page of 64 71 Page of 64 71

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 65 71 Page of 65 71

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt heavy_exclamation_mark

Page of 66 71 Page of 66 71

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert WARNING message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 67 71 Page of 67 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

Page of 68 71 Page of 68 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 69 71 Page of 69 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

Page of 70 71 Page of 70 71

Page of 71 71 Page of 71 71

Page 55: How to integrate the Elasticsearch Logstash Kibana (ELK ... · How to integrate the Elasticsearch Logstash Kibana (ELK) log analytics stack into IBM Bluemix Nick Cawood IBM Cloud

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert err message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_error in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

Page of 57 71 Page of 57 71

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 58 71 Page of 58 71

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 59 71 Page of 59 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert error message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

Page of 60 71 Page of 60 71

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

Page of 61 71 Page of 61 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 62 71 Page of 62 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt x

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert fatal message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

Page of 63 71 Page of 63 71

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

Page of 64 71 Page of 64 71

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 65 71 Page of 65 71

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt heavy_exclamation_mark

Page of 66 71 Page of 66 71

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert WARNING message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 67 71 Page of 67 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

Page of 68 71 Page of 68 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 69 71 Page of 69 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

Page of 70 71 Page of 70 71

Page of 71 71 Page of 71 71

Page 56: How to integrate the Elasticsearch Logstash Kibana (ELK ... · How to integrate the Elasticsearch Logstash Kibana (ELK) log analytics stack into IBM Bluemix Nick Cawood IBM Cloud

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 58 71 Page of 58 71

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 59 71 Page of 59 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert error message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

Page of 60 71 Page of 60 71

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

Page of 61 71 Page of 61 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 62 71 Page of 62 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt x

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert fatal message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

Page of 63 71 Page of 63 71

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

Page of 64 71 Page of 64 71

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 65 71 Page of 65 71

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt heavy_exclamation_mark

Page of 66 71 Page of 66 71

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert WARNING message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 67 71 Page of 67 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

Page of 68 71 Page of 68 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 69 71 Page of 69 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

Page of 70 71 Page of 70 71

Page of 71 71 Page of 71 71

Page 57: How to integrate the Elasticsearch Logstash Kibana (ELK ... · How to integrate the Elasticsearch Logstash Kibana (ELK) log analytics stack into IBM Bluemix Nick Cawood IBM Cloud

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 59 71 Page of 59 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert error message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

Page of 60 71 Page of 60 71

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

Page of 61 71 Page of 61 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 62 71 Page of 62 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt x

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert fatal message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

Page of 63 71 Page of 63 71

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

Page of 64 71 Page of 64 71

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 65 71 Page of 65 71

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt heavy_exclamation_mark

Page of 66 71 Page of 66 71

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert WARNING message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 67 71 Page of 67 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

Page of 68 71 Page of 68 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 69 71 Page of 69 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

Page of 70 71 Page of 70 71

Page of 71 71 Page of 71 71

Page 58: How to integrate the Elasticsearch Logstash Kibana (ELK ... · How to integrate the Elasticsearch Logstash Kibana (ELK) log analytics stack into IBM Bluemix Nick Cawood IBM Cloud

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert error message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt warning

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

Page of 60 71 Page of 60 71

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

Page of 61 71 Page of 61 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 62 71 Page of 62 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt x

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert fatal message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

Page of 63 71 Page of 63 71

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

Page of 64 71 Page of 64 71

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 65 71 Page of 65 71

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt heavy_exclamation_mark

Page of 66 71 Page of 66 71

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert WARNING message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 67 71 Page of 67 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

Page of 68 71 Page of 68 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 69 71 Page of 69 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

Page of 70 71 Page of 70 71

Page of 71 71 Page of 71 71

Page 59: How to integrate the Elasticsearch Logstash Kibana (ELK ... · How to integrate the Elasticsearch Logstash Kibana (ELK) log analytics stack into IBM Bluemix Nick Cawood IBM Cloud

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

Page of 61 71 Page of 61 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 62 71 Page of 62 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt x

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert fatal message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

Page of 63 71 Page of 63 71

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

Page of 64 71 Page of 64 71

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 65 71 Page of 65 71

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt heavy_exclamation_mark

Page of 66 71 Page of 66 71

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert WARNING message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 67 71 Page of 67 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

Page of 68 71 Page of 68 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 69 71 Page of 69 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

Page of 70 71 Page of 70 71

Page of 71 71 Page of 71 71

Page 60: How to integrate the Elasticsearch Logstash Kibana (ELK ... · How to integrate the Elasticsearch Logstash Kibana (ELK) log analytics stack into IBM Bluemix Nick Cawood IBM Cloud

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 62 71 Page of 62 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt x

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert fatal message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

Page of 63 71 Page of 63 71

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

Page of 64 71 Page of 64 71

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 65 71 Page of 65 71

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt heavy_exclamation_mark

Page of 66 71 Page of 66 71

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert WARNING message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 67 71 Page of 67 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

Page of 68 71 Page of 68 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 69 71 Page of 69 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

Page of 70 71 Page of 70 71

Page of 71 71 Page of 71 71

Page 61: How to integrate the Elasticsearch Logstash Kibana (ELK ... · How to integrate the Elasticsearch Logstash Kibana (ELK) log analytics stack into IBM Bluemix Nick Cawood IBM Cloud

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt x

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert fatal message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

Page of 63 71 Page of 63 71

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

Page of 64 71 Page of 64 71

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 65 71 Page of 65 71

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt heavy_exclamation_mark

Page of 66 71 Page of 66 71

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert WARNING message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 67 71 Page of 67 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

Page of 68 71 Page of 68 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 69 71 Page of 69 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

Page of 70 71 Page of 70 71

Page of 71 71 Page of 71 71

Page 62: How to integrate the Elasticsearch Logstash Kibana (ELK ... · How to integrate the Elasticsearch Logstash Kibana (ELK) log analytics stack into IBM Bluemix Nick Cawood IBM Cloud

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

Page of 64 71 Page of 64 71

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 65 71 Page of 65 71

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt heavy_exclamation_mark

Page of 66 71 Page of 66 71

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert WARNING message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 67 71 Page of 67 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

Page of 68 71 Page of 68 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 69 71 Page of 69 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

Page of 70 71 Page of 70 71

Page of 71 71 Page of 71 71

Page 63: How to integrate the Elasticsearch Logstash Kibana (ELK ... · How to integrate the Elasticsearch Logstash Kibana (ELK) log analytics stack into IBM Bluemix Nick Cawood IBM Cloud

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

Page of 65 71 Page of 65 71

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt heavy_exclamation_mark

Page of 66 71 Page of 66 71

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert WARNING message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 67 71 Page of 67 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

Page of 68 71 Page of 68 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 69 71 Page of 69 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

Page of 70 71 Page of 70 71

Page of 71 71 Page of 71 71

Page 64: How to integrate the Elasticsearch Logstash Kibana (ELK ... · How to integrate the Elasticsearch Logstash Kibana (ELK) log analytics stack into IBM Bluemix Nick Cawood IBM Cloud

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt heavy_exclamation_mark

Page of 66 71 Page of 66 71

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert WARNING message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 67 71 Page of 67 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

Page of 68 71 Page of 68 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 69 71 Page of 69 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

Page of 70 71 Page of 70 71

Page of 71 71 Page of 71 71

Page 65: How to integrate the Elasticsearch Logstash Kibana (ELK ... · How to integrate the Elasticsearch Logstash Kibana (ELK) log analytics stack into IBM Bluemix Nick Cawood IBM Cloud

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert WARNING message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt critical

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]

if [data1] == voicemailshowservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

Page of 67 71 Page of 67 71

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

Page of 68 71 Page of 68 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 69 71 Page of 69 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

Page of 70 71 Page of 70 71

Page of 71 71 Page of 71 71

Page 66: How to integrate the Elasticsearch Logstash Kibana (ELK ... · How to integrate the Elasticsearch Logstash Kibana (ELK) log analytics stack into IBM Bluemix Nick Cawood IBM Cloud

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == monverifyservicemanagement

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == ms-catalog-postpyretic-pomiculture

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == microservices-catalogapi-rstoner-1615

Page of 68 71 Page of 68 71

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 69 71 Page of 69 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

Page of 70 71 Page of 70 71

Page of 71 71 Page of 71 71

Page 67: How to integrate the Elasticsearch Logstash Kibana (ELK ... · How to integrate the Elasticsearch Logstash Kibana (ELK) log analytics stack into IBM Bluemix Nick Cawood IBM Cloud

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-UI-rstoner-1639

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

if [data1] == Microservices-OrdersAPI-rstoner-1622

pagerduty

event_type =gt trigger

description =gt data1 alert reported on host

details =gt

timestamp =gt timestamp

message =gt message

Page of 69 71 Page of 69 71

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

Page of 70 71 Page of 70 71

Page of 71 71 Page of 71 71

Page 68: How to integrate the Elasticsearch Logstash Kibana (ELK ... · How to integrate the Elasticsearch Logstash Kibana (ELK) log analytics stack into IBM Bluemix Nick Cawood IBM Cloud

service_key =gt ltpagerduty_service_keygt

incident_key =gt logstashhosttype

slack

url =gt ltSlackIntegrationURLgt

channel =gt slack_channel

username =gt slack_user

icon_emoji =gt interrobang

icon_url =gt [icon url would be overriden by icon_emoji - optional]

format =gt ELK alert Stopped message reported for data1 on host message

attachments =gt

http

url =gt httpltOmnibusIPgtltOmnibusPortgt

http_method =gt post

format =gt json

syslog

facility =gt log alert

host =gt ltOmnibusIPgt

port =gt ltOmnibusPortgt

severity =gt notice

appname =gt logstash

message =gt msgtosend

msgid =gt tstamp

procid =gt data1

rfc =gt rfc5424

sourcehost =gt host

Page of 70 71 Page of 70 71

Page of 71 71 Page of 71 71

Page 69: How to integrate the Elasticsearch Logstash Kibana (ELK ... · How to integrate the Elasticsearch Logstash Kibana (ELK) log analytics stack into IBM Bluemix Nick Cawood IBM Cloud

Page of 71 71 Page of 71 71


Top Related