How to integrate the Elasticsearch Logstash
Kibana (ELK) log analytics stack into IBM Bluemix
Nick Cawood IBM Cloud Client Adoption and Technical Enablement
Client and Technical Engagement13 August 2016
Page of 1 71
CONTENTS13
Introducon13 313
What13 is13 Elascsearch13 Logstash13 Kibana13 313
LOGSTASH 3 ELASTICSEARCH 4 KIBANA 4 ELK SYSTEM ARCHITECTURE 4
Service13 Management13 architecture13 613
Log13 Monitoring13 System13 architecture13 613
Bluemix13 environment13 system13 architecture13 613
Creang13 a13 virtual13 server13 in13 Bluemix13 713
CREATE AN SSH KEY 8 PROVISIONING A VIRTUAL SERVER 9 SET UP THE VIRTUAL SERVER 12
Log on to the virtual server 12 Acces the root user 12 Copy files to the virtual server 13 Install and access the virtual server via VNC server 13
Install13 the13 ELK13 stack13 1313
INSTALL ELASTICSEARCH 14 Start Elasticsearch 14
INSTALL KIBANA 15 Start Kibana 18
BROADCAST BLUEMIX APP LOG RECORDS 19 INSTALL LOGSTASH 19
Write Logstash configuration 19 Start Logstash 21 Monitor Logstash processing 21
Use13 ELK13 2213
KIBANA DISCOVER PAGE 22 Change the time filter 22 Custom search queries 23 Saved searches 24
BUILD SIMPLE KIBANA HISTOGRAMS 27 Alerng13 with13 ELK13 3613
CREATE ALERTS IN LOGSTASH 36 ENRICH ALERTS WITH MESSAGE DETAIL 38
Grok 38 Translate 38
ADD CUSTOM PLUGINS TO LOGSTASH 39 Filter translate plugin 39 Output Slack plugin 39 Output syslog plugin 40
INTEGRATE LOGSTASH WITH SLACK 40 INTEGRATE LOGSTASH WITH PAGERDUTY 45 INTEGRATE LOGSTASH WITH IBM NETCOOL OMNIBUS 48
Appendix13 Logstash13 configuraon13 51
Page of 2 71
Introduc0on13 IBMreg13 Bluemixreg13 IBMrsquos13 PlaRorm13 as13 a13 Service13 (PaaS)13 allows13 you13 to13 rapidly13 build13 and13 deploy13 an13 applicaon13 Applicaons13 can13 run13 in13 a13 nave13 or13 a13 hybrid13 Bluemix13 environment13 13 13
This13 document13 describes13 the13 integraon13 of13 the13 Elascsearch13 Logstash13 Kibana13 (ELK)13 log13 analycs13 open13 source13 stack13 into13 the13 Bluemix13 environment13 how13 to13 collect13 logs13 from13 Bluemix13 applicaons13 (apps)13 and13 send13 alerts13 from13 those13 log13 records13 to13 selected13 alertevent13 noficaon13 targets13 (PagerDuty13 Slack13 and13 IBM13 Netcool13 Omnibus)13
This13 document13 describes13 how13 to13 provision13 Bluemix13 virtual13 servers13 in13 the13 Bluemix13 environment13 install13 the13 ELK13 stack13 on13 a13 Bluemix13 virtual13 server13 and13 configure13 the13 ELK13 stack13 to13 integrate13 with13 other13 components13 deployed13 on13 the13 Bluemix13 environment13
The13 steps13 to13 integrate13 the13 ELK13 stack13 into13 the13 Cloud13 Architecture13 and13 Soluon13 Engineering13 (CASE)13 environment13 are13 used13 as13 the13 basis13 for13 this13 document13 13 For13 reference13 both13 the13 ELK13 architecture13 within13 the13 CASE13 environment13 and13 the13 whole13 CASE13 environment13 are13 defined13 using13 system13 architecture13 diagrams13 as13 well13 as13 diagrams13 defining13 the13 ELK13 structure13 and13 ELK13 integraon13 into13 Bluemix13 architecture13
What13 is13 Elas0csearch13 Logstash13 Kibana13 Elascsearch13 Logstash13 Kibana13 is13 an13 open13 source13 stack13 of13 three13 applicaons13 that13 work13 together13 to13 provide13 an13 end-shy‐to-shy‐end13 search13 and13 visualisaon13 plaRorm13 for13 the13 invesgaon13 of13 log13 file13 sources13 in13 real13 me13 Log13 file13 records13 can13 be13 searched13 from13 a13 variety13 of13 log13 sources13 charts13 and13 dashboards13 built13 to13 visualize13 the13 log13 records13 The13 three13 applicaons13 are13 Elascsearch13 Logstash13 and13 Kibana13
Logstash13 Logstash13 is13 an13 applicaon13 that13 allows13 the13 ELK13 stack13 to13 manage13 the13 gathering13 transformaon13 and13 transport13 of13 log13 file13 records13 Logstash13 is13 able13 to13 pull13 records13 from13 remote13 sources13 or13 listen13 for13 records13 on13 specified13 ports13
Logstash13 has13 three13 areas13 of13 funcon13
bull Process13 input13 of13 the13 log13 file13 records13 from13 the13 remote13 source13 You13 can13 configure13 Logstash13 to13 receive13 records13 in13 a13 variety13 of13 methods13 for13 example13 reading13 from13 a13 flat13 file13 or13 from13 a13 TCP13 port13 or13 direct13 from13 a13 remote13 or13 local13 syslog13 These13 methods13 are13 defined13 in13 the13 Logstash13 online13 support13 and13 input13 secon13 of13 the13 Logstash13 conf13 file13 Once13 all13 input13 processing13 is13 completed13 Logstash13 moves13 onto13 the13 next13 funcon13
bull Transform13 and13 enrich13 the13 log13 records13 that13 take13 place13 in13 the13 Logstash13 filter13 secon13 Any13 number13 of13 changes13 to13 the13 format13 (and13 content13 if13 required)13 of13 the13 log13 file13 record13 can13 be13 made13 using13 the13 various13 methods13 provided13 with13 Logstash13 for13 example13 methods13 to13 add13 and13 enrich13 log13 record13 fields13 and13 the13 flagging13 I13 describe13 excluding13 and13 liming13 records13 for13 alerng13 later13 in13 this13 document13 Aaer13 the13 log13 record13 is13 transformed13 Logstash13 processes13 the13 record13
bull Send13 the13 log13 records13 to13 the13 Elascsearch13 applicaon13 from13 the13 output13 secon13 of13 the13 Logstash13 configuraon13 A13 specific13 elascsearch13 method13 is13 included13 with13 the13 Logstash13 instance13 that13 sends13 the13 log13 records13 to13 the13 Elascsearch13 instance13 However13 as13 with13 the13 input13 and13 filter13
Page of 3 71 Page of 3 71
secons13 a13 variety13 of13 methods13 are13 available13 to13 process13 the13 data13 sent13 from13 the13 filter13 secon13 For13 example13 the13 records13 can13 be13 sent13 to13 external13 alerng13 tools13 such13 as13 PagerDuty13 social13 media13 tools13 such13 as13 Slack13 or13 event13 management13 hubs13 like13 IBM13 Netcool13 Omnibus13 via13 output13 methods13 syslog13 or13 hcp13 13
As13 an13 open13 source13 tool13 non-shy‐Logstash13 employees13 develop13 other13 methods13 and13 share13 them13 with13 the13 user13 community13 for13 addion13 to13 the13 library13 of13 methods13 Logstash13 can13 execute13 (examples13 of13 this13 process13 are13 included13 later13 in13 this13 document)13
Elas0csearch13 Elascsearch13 is13 the13 index13 store13 and13 query13 applicaon13 on13 the13 ELK13 stack13 It13 provides13 a13 standard13 full13 text13 search13 facility13 of13 the13 log13 file13 records13 processed13 by13 Logstash13 The13 ELK13 online13 help13 documentaon13 describes13 Elascsearch13 as13 follows13 ldquoIt13 is13 generally13 used13 as13 the13 underlying13 enginetechnology13 that13 powers13 applicaons13 that13 have13 complex13 search13 features13 and13 requirementsrdquo13
Kibana13 Kibana13 is13 the13 user13 interface13 for13 the13 ELK13 stack13 Kibana13 integrates13 with13 Elascsearch13 to13 visualise13 the13 results13 of13 searches13 and13 allow13 the13 creaon13 of13 charts13 and13 dashboards13 based13 on13 the13 searches13 The13 searches13 charts13 and13 dashboard13 can13 visualise13 real-shy‐me13 or13 historical13 log13 records13 Users13 access13 Kibana13 through13 an13 hcp13 URL13 from13 a13 client13 web13 browser13
ELK13 System13 Architecture13 The13 ELK13 stack13 can13 be13 described13 as13 follows13
Page of 4 71 Page of 4 71
13
Page of 5 71 Page of 5 71
Service13 Management13 architecture13 The13 IBM13 Cloud13 Architecture13 Center13 created13 a13 standard13 reference13 architecture13 for13 how13 to13 set13 up13 your13 cloud13 environemnt13 to13 handle13 incident13 management13 13
View13 the13 standard13 Service13 Management13 architecure13
13 13
Log13 Monitoring13 System13 architecture13 The13 following13 diagram13 shows13 a13 standard13 architecture13 for13 the13 log-shy‐monitoring13 system13 described13 in13 this13 tutorial13 13
13
Bluemix13 environment13 system13 architecture13 The13 ELK13 stack13 can13 be13 installed13 into13 a13 variety13 of13 environments13 When13 using13 Bluemix13 the13 following13 figure13 shows13 what13 the13 environment13 might13 look13 like13
Page of 6 71 Page of 6 71
13
The13 Bluemix13 cloud13 plaRorm13 offers13 a13 Public13 network13 and13 access13 to13 US13 South13 and13 UK13 environments13 From13 all13 these13 environments13 third-shy‐party13 cloud-shy‐based13 soluons13 are13 also13 accessible13
The13 ELK13 stack13 is13 installed13 in13 the13 Bluemix13 United13 Kingdom13 environment13 because13 this13 is13 the13 environment13 where13 Bluemix13 has13 virtual13 servers13 available13 for13 provisioning13 Other13 applicaons13 in13 the13 CASE13 project13 were13 also13 installed13 on13 virtual13 servers13 in13 the13 Bluemix13 United13 Kingdom13 region13
Bluemix13 apps13 were13 used13 to13 create13 a13 system13 to13 be13 monitored13 by13 ELK13 These13 apps13 were13 hosted13 in13 the13 Bluemix13 US13 South13 environment13 using13 Bluemix13 microservices13 which13 produce13 logs13 during13 normal13 operaon13 and13 during13 crashes13 and13 outages13
Through13 the13 Bluemix13 Cloud13 Foundry13 process13 the13 log13 records13 can13 be13 broadcast13 from13 the13 Bluemix13 apps13 so13 Logstash13 can13 receive13 them13 and13 send13 them13 to13 Elascsearch13 within13 the13 ELK13 stack13 or13 other13 virtual13 servers13 (hosng13 other13 IBM13 or13 third-shy‐party13 products)13 in13 the13 Bluemix13 United13 Kingdom13 environment13 or13 to13 third-shy‐party13 applicaons13 in13 the13 public13 cloud13 (Pager13 Duty13 Slack13 etc)13
Crea0ng13 a13 virtual13 server13 in13 Bluemix13 With13 Bluemix13 you13 can13 provision13 a13 virtual13 server13 in13 the13 cloud13 to13 install13 standalone13 or13 integrated13 system13 components13 Not13 all13 Bluemix13 regions13 offer13 this13 service13 all13 the13 me13 due13 to13 beta-shy‐project13 constraints13 In13 this13 arclersquos13 examples13 virtual13 servers13 were13 allowed13 to13 be13 provisioned13 in13 the13 United13 Kingdom13 region13 For13 more13 informaon13 read13 how13 to13 provision13 a13 Bluemix13 virtual13 server13 including13 some13 simple13 setup13 steps13 for13 ease13 of13 use13
Page of 7 71 Page of 7 71
Create13 an13 SSH13 key13 To13 access13 your13 virtual13 server13 once13 it13 is13 provisioned13 allocate13 an13 SSH13 key13 to13 the13 virtual13 server13
To13 generate13 a13 publicprivate13 RSA13 key13 pair13 follow13 these13 steps13
1 In13 a13 UNIX13 or13 Linuxreg13 window13 run13 the13 following13 command13 specifying13 a13 ltuniquekeynamegt13 of13 your13 your13 choice13 and13 a13 password13 or13 passphrase13 (memorize13 the13 password)$13 ssh-shy‐keygen13 -shy‐t13 rsa13 -shy‐f13 ltuniquekeynamegt13
2 Enter13 the13 passphrase13 (empty13 for13 no13 passphrase)13
3 Enter13 the13 same13 passphrase13 again13
Your13 idenficaon13 has13 been13 saved13 in13 ltuniquekeynamegt13
Your13 public13 key13 has13 been13 saved13 in13 ltuniquekeynamegtpub13
The13 key13 fingerprint13 is13
d6268464506525930ce4632c693c203c13 ltusergtltservergt13
The13 keys13 randomart13 image13 is13
+--[ RSA 2048]----+
|o ++=o |
| E o ooo |
| |
| + o |
| S o |
| o |
| |
| |
| |
+-----------------+
Two13 keys13 are13 created13 public13 and13 private13 The13 public13 key13 contents13 are13 used13 to13 create13 a13 virtual13 server13 with13 a13 public13 IP13 address13 The13 private13 key13 is13 used13 to13 access13 the13 virtual13 server13 from13 a13 client13 machine13
Page of 8 71 Page of 8 71
Provisioning13 a13 virtual13 server13 1 A13 Bluemix13 account13 is13 required13 for13 this13 process13 Sign13 up13 here13 and13 log13 in13 Make13 sure13 your13
environment13 is13 correct13 For13 example13 if13 you13 are13 in13 the13 United13 Kingdom13 select13 the13 United13 Kingdom13 region13 in13 the13 Bluemix13 console13
13
13
2 Add13 a13 virtual13 server13 to13 your13 Bluemix13 environment13 Click13 Run13 Virtual13 Servers13
13
3 Select13 an13 OS13 image13 from13 the13 EXISTING13 list13 For13 ELK13 select13 Centos13 OS13 (use13 latest13 available13 version)13
Page of 9 71 Page of 9 71
4 Name13 the13 virtual13 server13
5 Make13 sure13 Assign13 public13 IP13 address13 is13 checked13
13
6 Click13 Add13 Key13 to13 add13 the13 SSH13 key13 you13 created13 previously13 Copy13 the13 contents13 of13 the13 public13 key13 into13 the13 Public13 Key13 to13 import13 field13
Page of 10 71 Page of 10 71
13
7 Click13 OK13 Then13 click13 CREATE13
The13 virtual13 server13 is13 provisioned13 and13 added13 to13 the13 Bluemix13 dashboard13 under13 VIRTUAL13 SERVERS13
Page of 11 71 Page of 11 71
13
Set13 up13 the13 virtual13 server13
Log13 on13 to13 the13 virtual13 server13 Now13 you13 can13 log13 on13 to13 the13 virtual13 server13 and13 add13 new13 users13 and13 programs13 to13 make13 it13 easier13 to13 access13 the13 virtual13 server13
To13 access13 the13 server13 go13 to13 the13 UNIX13 or13 Linux13 window13 that13 you13 used13 to13 create13 the13 SSH13 key13 In13 the13 directory13 where13 the13 ltuniquekeynamegt13 files13 exist13 run13 the13 following13 command13 referencing13 the13 SSH13 keyname13 the13 public13 IP13 address13 Bluemix13 allocated13 for13 the13 virtual13 server13 and13 the13 password13 for13 the13 SSH13 key13
[userserver ~]$ ssh -i ltuniquekeynamegt ibmcloudltpublicIPaddressgt
Enter13 the13 passphrase13 for13 the13 key13 ltuniquekeynamegt13
Last login Tue Jul 5 125546 2016 from 108-210-96-231lightspeedrlghncsbcglobalnet
[ibmcloudelkforcase ~]$
ibmcloud13 is13 the13 default13 user13 for13 the13 Bluemix13 virtual13 servers13 and13 allows13 SSH13 to13 access13 the13 server13
Acces13 the13 root13 user13 To13 access13 the13 root13 user13 log13 on13 as13 ibmcloud13 user13 and13 run13 the13 following13 command13
[ibmcloudelkforcase ~]$ sudo bash
[rootelkforcase ibmcloud]
The13 root13 user13 can13 be13 used13 to13 add13 programs13 to13 the13 server13 and13 create13 and13 edit13 users13 as13 required13 to13 access13 applicaons13 installed13 on13 the13 virtual13 server13
Page of 12 71 Page of 12 71
Copy13 files13 to13 the13 virtual13 server13 To13 move13 installaon13 files13 to13 the13 server13 go13 to13 the13 UNIX13 or13 Linux13 window13 you13 used13 to13 create13 the13 SSH13 key13 and13 the13 directory13 where13 the13 ltuniquekeynamegt13 files13 exist13 Move13 the13 installaon13 files13 to13 the13 same13 locaon13
Run13 the13 following13 command13 referencing13 the13 SSH13 keyname13 the13 public13 IP13 address13 Bluemix13 allocated13 for13 the13 virtual13 server13 and13 the13 password13 for13 the13 SSH13 key13
[userserver ~]$ scp ndashi ltuniquekeynamegt ltINSTALLATIONFILENAMEgt ibmcloudltpublicIPaddressgthomeibmcloud
Enter13 the13 passphrase13 for13 key13 cloudkey313
This13 copies13 the13 file13 using13 the13 ibmcloud13 user13 to13 the13 homeibmcloud13 directory13 on13 the13 virtual13 server13
Install13 and13 access13 the13 virtual13 server13 via13 VNC13 server13 Using13 the13 ssh13 command13 to13 access13 the13 virtual13 server13 is13 not13 the13 most13 stable13 method13 and13 does13 not13 offer13 X-shy‐Windows13 support13 Therefore13 we13 suggest13 that13 you13 use13 a13 VNC13 client13 to13 access13 the13 virtual13 server13
Installing13 a13 VNC13 client13 should13 be13 done13 as13 the13 root13 user13 Use13 the13 yum13 command13 as13 follows13
[ibmcloudelkforcase ~]$ yum groupinstall GNOME Desktop
[ibmcloudelkforcase ~]$ yum install tigervnc-server
These13 commands13 first13 install13 a13 desktop13 style13 OS13 user13 interface13 and13 then13 a13 VNC13 client13 to13 remotely13 access13 the13 desktop13 style13 OS13
Then13 as13 either13 the13 root13 user13 or13 another13 user13 (example13 below13 uses13 elk)13 run13 the13 following13 command13 to13 create13 the13 VNC13 client13 connecon13
[elkelkforcase ~]$ vncserver
New elkforcase2 (elk) desktop is elkforcase2
Starting applications specified in homeelkvncxstartup
Log file is homeelkvncelkforcase2log
[elkelkforcase ~]$
Install13 a13 VNC13 client13 on13 the13 client13 machine13 used13 to13 access13 the13 Internet13 and13 (in13 the13 example13 above)13 use13 the13 command13 ltpublicIPadddressgt213 and13 the13 password13 for13 ELK13 to13 access13 the13 virtual13 server13
Using13 a13 VNC13 client13 means13 that13 you13 can13 leave13 long13 duraon13 installaons13 running13 without13 fear13 of13 the13 terminal13 losing13 connecon13 and13 the13 installaons13 failing13
Install13 the13 ELK13 stack13 Download13 the13 source13 soaware13 code13 for13 the13 ELK13 stack13 from13 the13 ELK13 website13 13
At13 the13 me13 of13 wring13 the13 downloaded13 soaware13 files13 were13
Page of 13 71 Page of 13 71
13
Transfer13 the13 files13 to13 the13 virtual13 server13 (using13 the13 scp13 method13 detailed13 above)13
Install13 Elas0csearch13 As13 a13 root13 user13 create13 a13 non-shy‐root13 user13 with13 the13 adduser13 command13
Create13 a13 directory13 to13 install13 Elascsearch13 into13 and13 untar13 the13 contents13 of13 the13 Elascsearch13 install13 file13 into13 that13 directory13
[elkelkforcase]$ cd home
[elkelkforcase]$ mkdir elasticsearch
[elkelkforcase]$ cp homeibmcloudelasticsearch-233targz homeelasticsearch
[elkelkforcase]$ cd elasticsearch
[elkelkforcase]$ gunzip elasticsearch-233targz
[elkelkforcase]$ untar elasticsearch-233tar
Elascsearch13 is13 now13 installed13
Start13 Elas0csearch13 To13 start13 Elascsearch13 go13 to13 the13 bin13 directory13 as13 the13 non-shy‐root13 user13 when13 you13 installed13 it13 and13 run13 the13 elascsearch13 executable13
[elkelkforcase]$ cd homeelasticsearchelasticsearch-233bin
[elkelkforcase]$ elasticsearch amp
[2] 22211
[elkelkforcase]$ [2016-07-05 133857730][INFO ][node] [Arsenal] version[233] pid[22211] build[218bdf12016-05-17T154004Z]
[2016-07-05 133857740][INFO ][node] [Arsenal] initializing
[2016-07-05 133858984][INFO ][plugins] [Arsenal] modules [reindex lang-expression lang-groovy] plugins [] sites []
[2016-07-05 133859146][INFO ][env] [Arsenal] using [1] data paths mounts [[ (rootfs)]] net usable_space [155gb] net total_space [199gb] spins [unknown] types [rootfs]
[2016-07-05 133859146][INFO ][env] [Arsenal] heap size [10156mb] compressed ordinary object pointers [true]
Page of 14 71 Page of 14 71
[2016-07-05 133859147][WARN ][env] [Arsenal] max file descriptors [4096] for elasticsearch process likely too low consider increasing to at least [65536]
[2016-07-05 133903526][INFO ][node] [Arsenal] initialized
[2016-07-05 133903526][INFO ][node] [Arsenal] starting
To13 test13 that13 Elascsearch13 is13 running13 correctly13 open13 the13 browser13 in13 the13 VNC13 client13 and13 run13 localhost920013
13
Install13 Kibana13 ELK13 recommends13 installing13 and13 running13 Kibana13 as13 root13 13
1 As13 root13 user13 create13 an13 installaon13 directory13 and13 untar13 the13 installaon13 file13
[rootelkforcase]$ cd home
[rootelkforcase]$ mkdir kibana
[rootelkforcase]$ cp homeibmcloudkibana-451-linux-x64targz homekibana
[rootelkforcase]$ cd kibana
[rootelkforcase]$ gunzip kibana-451-linux-x64targz
[rootelkforcase]$ untar kibana-451-linux-x64tar
2 Modify13 the13 manifestyml13 configuraon13 file13 Go13 to13 the13 kibana-shy‐451-shy‐linux-shy‐x6413 directory13
[rootelkforcase]$ cd homekibanakibana-451-linux-x64
3 Edit13 the13 manifestyml13 file13 so13 the13 contents13 are13
applicaons13
-shy‐ name13 kibana-shy‐in-shy‐bluemix13
memory13 128M13
host13 kibana-shy‐in-shy‐bluemix13
buildpack13 hcpsgithubcomcloudfoundry-shy‐communitynginx-shy‐buildpackgit13
Page of 15 71 Page of 15 71
4 Edit13 the13 configkibanayml13 file13 so13 the13 contents13 are13
Kibana is served by a back end server This controls which port to use
port 5601
The host to bind the server to
host 0000
If you are running kibana behind a proxy and want to mount it at a path
specify that path here The basePath cant end in a slash
serverbasePath
The maximum payload size in bytes on incoming server requests
servermaxPayloadBytes 1048576
The Elasticsearch instance to use for all your queries
elasticsearchurl httplocalhost9200
preserve_elasticsearch_host true will send the hostname specified in `elasticsearch` If you set it to false
then the host you use to connect to this Kibana instance will be sent
elasticsearchpreserveHost true
Kibana uses an index in Elasticsearch to store saved searches visualizations
and dashboards It will create a new index if it doesnt already exist
kibanaindex kibana
The default application to load
kibanadefaultAppId discover
If your Elasticsearch is protected with basic auth these are the user credentials
used by the Kibana server to perform maintenance on the kibana_index at startup Your
Kibana users will still need to authenticate with Elasticsearch (which is proxied through
the Kibana server)
Page of 16 71 Page of 16 71
elasticsearchusername user
elasticsearchpassword pass
SSL for outgoing requests from the Kibana Server to the browser (PEM formatted)
serversslcert pathtoyourservercrt
serversslkey pathtoyourserverkey
Optional setting to validate that your Elasticsearch backend uses the same key files (PEM formatted)
elasticsearchsslcert pathtoyourclientcrt
elasticsearchsslkey pathtoyourclientkey
If you need to provide a CA certificate for your Elasticsearch instance put
the path of the pem file here
elasticsearchsslca pathtoyourCApem
Set to false to have a complete disregard for the validity of the SSL
certificate
elasticsearchsslverify true
Time in milliseconds to wait for elasticsearch to respond to pings defaults to
request_timeout setting
elasticsearchpingTimeout 1500
Time in milliseconds to wait for responses from the back end or elasticsearch
This must be gt 0
elasticsearchrequestTimeout 30000
Time in milliseconds for Elasticsearch to wait for responses from shards
Set to 0 to disable
elasticsearchshardTimeout 0
Time in milliseconds to wait for Elasticsearch at Kibana startup before retrying
elasticsearchstartupTimeout 5000
elasticsearchstartupTimeout 30000
Page of 17 71 Page of 17 71
Set the path to where you would like the process id file to be created
pidfile varrunkibanapid
If you would like to send the log output to a file you can set the path below
loggingdest stdout
Set this to true to suppress all logging output
loggingsilent false
Set this to true to suppress all logging output except for error messages
loggingquiet false
Set this to true to log all events including system usage information and all requests
loggingverbose false
Kibana13 is13 now13 installed13
Start13 Kibana13 To13 start13 Kibana13 go13 to13 the13 bin13 directory13 and13 run13 the13 kibana13 executable13
[rootelkforcase]$
[rootelkforcase]$ cd homekibanakibana-451-linux-x64bin
[rootelkforcase]$ kibana amp
[1] 22283
[rootelkforcase] log [095900542] [info][status][pluginkibana] Status changed from uninitialized to green - Ready
log [095900600] [info][status][pluginelasticsearch] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [095900689] [info][status][pluginkbn_vislib_vis_types] Status changed from uninitialized to green - Ready
log [095900717] [info][status][pluginmarkdown_vis] Status changed from uninitialized to green - Ready
log [095900721] [info][status][pluginmetric_vis] Status changed from uninitialized to green - Ready
log [095900814] [info][status][pluginspyModes] Status changed from uninitialized to green - Ready
log [095900934] [info][status][pluginstatusPage] Status changed from uninitialized to green - Ready
Page of 18 71 Page of 18 71
log [095900941] [info][status][plugintable_vis] Status changed from uninitialized to green - Ready
log [095901067] [info][listening] Server running at http00005601
log [095906083] [info][status][pluginelasticsearch] Status changed from yellow to yellow - No existing Kibana index found
log [095908942] [info][status][pluginelasticsearch] Status changed from yellow to green - Kibana index ready
Kibana13 is13 now13 started13
Broadcast13 Bluemix13 app13 log13 records13 Bluemix13 apps13 (which13 are13 shown13 in13 the13 US13 South13 Bluemix13 region13 of13 the13 System13 Architecture13 diagram)13 will13 produce13 logs13 in13 a13 syslog13 rfc542413 format13 using13 a13 Bluemix13 tool13 called13 Loggregator13
For13 Logstash13 to13 receive13 these13 logs13 each13 Bluemix13 app13 needs13 to13 broadcast13 these13 logs13 To13 do13 that13 run13 the13 following13 commands13 on13 each13 Bluemix13 app13 using13 the13 public13 IP13 address13 of13 the13 ELK13 virtual13 server13 the13 ELK13 tag13 to13 trace13 the13 process13 and13 insert13 the13 name13 of13 each13 app13 in13 turn13
cf cups elk -l syslogltpublicIPaddress5000 cf bind-service ltmyappgt elk
cf restage ltmyappgt
Install13 Logstash13 To13 install13 Logstash13 as13 root13 user13 create13 a13 directory13 for13 the13 installaon13 and13 untar13 the13 installaon13 file13 there13
13 13 [rootelkforcase]$ cd home
[rootelkforcase]$ mkdir logstash
[rootelkforcase]$ cp homeibmcloudlogstash-233targz homelogstash
[rootelkforcase]$ cd logstash
[rootelkforcase]$ gunzip logstash-233targz
[rootelkforcase]$ untar logstash-233tar
Write13 Logstash13 configura0on13 Logstash13 requires13 a13 configuraon13 file13 to13 be13 wricen13 to13 receive13 transform13 and13 output13 the13 log13 records13 to13 Elascsearch13 This13 secon13 explains13 the13 basic13 requirements13 of13 processing13 Bluemix13 app13 log13 records13 into13 Elascsearch13 (more13 complex13 configuraons13 are13 detailed13 later13 in13 this13 document13 for13 the13 integraons13 with13 third-shy‐party13 tools)13
Create13 a13 conf13 file13 as13 root13 user13
[rootelkforcase]$ cd homelogstashlogstash-233
[rootelkforcase]$ mkdir config
Page of 19 71 Page of 19 71
[rootelkforcase]$ cd config
[rootelkforcase]$ touch loggregatorconf
Edit13 the13 loggregator13 file13 with13 the13 following13 contents13
13 Input13 secon13 defines13 which13 log13 records13 to13 receive13
13 port13 and13 type13 reflect13 method13 selected13 for13 Cloud13 Foundry13 to13 broadcast13 the13 records13
input
tcp
port =gt 5000
type =gt syslog
13 Filter13 secon13 ndash13 transforms13 the13 data13
filter
Add a field to identify the records as Cloud Foundry based
mutate
add_field =gt format =gt cf
13 Output13 secon13 ndash13 sends13 the13 data13 to13 its13 desnaon(s)13
output
13 Use13 the13 elascsearch13 plugin13 to13 send13 the13 data13 to13 the13 elascsearch13 instance13
13 elasticsearch
hosts =gt localhost
stdout codec =gt rubydebug
13 End13 of13 Config13
Save13 the13 file13 Verify13 the13 format13 of13 the13 file13 is13 acceptable13 by13 running13 the13 following13 commands13
[rootelkforcase]$ cd homelogstashlogstash-233
[rootelkforcase]$ binlogstash agent --configtest -f configloggregatorconf
Page of 20 71 Page of 20 71
Configuraon13 is13 OK13
Start13 Logstash13 Aaer13 you13 have13 wricen13 and13 verified13 the13 Logstash13 configuraon13 you13 can13 start13 Logstash13 to13 process13 records13 from13 the13 Bluemix13 apps13 (via13 Cloud13 Foundry)13 Run13 the13 following13 as13 root13 user13
[rootelkforcase]$ cd homelogstashlogstash-233
[rootelkforcase]$ binlogstash agent -f configloggregatorconf
Logstash13 is13 now13 started13
Monitor13 Logstash13 processing13 Once13 Logstash13 is13 running13 and13 log13 records13 are13 being13 produced13 by13 the13 Bluemix13 apps13 you13 can13 monitor13 progress13 in13 the13 same13 window13 that13 was13 used13 to13 start13 Logstash13 Output13 is13 similar13 to13 the13 following13
13
The13 log13 records13 being13 broadcast13 from13 the13 Bluemix13 apps13 are13 now13 sent13 to13 Elascsearch13 from13 the13 Logstash13 output13 configuraon13 Records13 are13 sent13 in13 real13 me13 If13 Logstash13 is13 not13 running13 while13 log13 records13 are13 being13 produced13 then13 these13 records13 will13 not13 be13 processed13 by13 Logstash13 and13 will13 not13 be13 stored13 in13 Elascsearch13
Page of 21 71 Page of 21 71
Use13 ELK13 The13 log13 records13 from13 the13 Bluemix13 apps13 are13 now13 being13 stored13 and13 indexed13 by13 the13 Elascsearch13 component13 This13 data13 can13 be13 accessed13 via13 the13 Kibana13 applicaon13
Kibana13 Discover13 page13 Using13 a13 web13 browser13 either13 directly13 from13 a13 client13 or13 using13 the13 VNC13 client13 access13 Kibana13 using13 the13 publicIPaddress13 of13 the13 ELK13 virtual13 server13
hcpltpublicIPaddressgt560113
No13 userpassword13 credenals13 are13 required13
Kibana13 defaults13 to13 a13 search13 of13 all13 records13 ()13 in13 the13 last13 1513 minutes13 and13 returns13 the13 results13
13
More13 complex13 searches13 are13 possible13 by13 changing13 the13 me13 filter13 and13 supplying13 search13 queries13
Change13 the13 0me13 filter13 To13 change13 the13 me13 ranges13 click13 the13 clock13 symbol13 in13 the13 top-shy‐right13 of13 the13 Kibana13 Discover13 page13
13
You13 will13 see13 the13 following13 opons13
13
Page of 22 71 Page of 22 71
A13 variety13 of13 quick13 me13 ranges13 can13 be13 selected13 or13 you13 can13 switch13 to13 relave13 or13 absolute13 me13 ranges13
Quick13 me13 ranges13 allow13 switching13 between13 last13 X13 minutes13 to13 last13 Y13 hours13 or13 Z13 days13 etc13
Relave13 me13 ranges13 allow13 more13 specific13 versions13 of13 the13 Quick13 opons13 from13 specific13 minuteshoursdays13 to13 now13
13
Absolute13 me13 ranges13 specify13 the13 start13 and13 end13 mes13 for13 the13 searches13
13
Custom13 search13 queries13 You13 can13 use13 the13 text13 window13 of13 the13 Discover13 page13 to13 search13 for13 matches13 within13 the13 log13 records13 You13 can13 use13 Apache13 Lucene13 or13 Elascsearch13 query13 DSL13 syntax13
13
Page of 23 71 Page of 23 71
Saved13 searches13 You13 can13 save13 any13 custom13 search13 syntax13 and13 choice13 of13 me13 range13 for13 future13 ease13 of13 use13 Any13 relave13 or13 absolute13 me13 frame13 is13 saved13 as13 part13 of13 the13 search13 13
To13 save13 a13 search13
1 Set13 up13 and13 run13 the13 required13 syntax13 and13 me13 frame13
13
2 Click13 the13 Save13 Search13 icon13
13
3 Type13 in13 a13 name13 for13 the13 saved13 search13
Page of 24 71 Page of 24 71
13
7 To13 delete13 or13 update13 saved13 searches13 click13 13 manage13 saved13 searches13
13
Use13 the13 edit13 icons13 to13 make13 any13 changes13
Page of 26 71 Page of 26 71
13
Build13 simple13 Kibana13 histograms13 Kibana13 allows13 you13 to13 build13 charts13 and13 dashboards13 based13 on13 the13 search13 capability13 offered13 by13 the13 Discover13 page13 Saved13 searches13 can13 be13 used13 as13 the13 basis13 for13 charts13 and13 dashboards13 built13 using13 those13 charts13 To13 build13 a13 simple13 histogram13
1 Select13 the13 Visualize13 page13
13
2 Select13 Ver0cal13 Bar13 Chart13
13
3 To13 use13 a13 saved13 search13 click13 From13 a13 saved13 search13
Page of 27 71 Page of 27 71
13
4 Select13 a13 previously13 saved13 search13
13
5 Select13 to13 edit13 the13 x-shy‐axis
6 Edit13 the13 fields13 with13 informaon13 about13 the13 aggregaon13 field13 interval13 and13 custom13 label13
Page of 28 71 Page of 28 71
13
Page of 29 71 Page of 29 71
7 Click13 the13 green13 arrow13 The13 histogram13 is13 created13 and13 run13 based13 on13 the13 saved13 search
13
8 To13 save13 the13 histogram13 click13 the13 Save13 Visualiza0on13 icon13 and13 add13 a13 tle13
13
13
9 Click13 Save13
10 To13 view13 the13 visualisaon13 click13 the13 Dashboard13 page13
Page of 30 71 Page of 30 71
13
11 Click13 the13 Add13 Visualiza0on13 icon13
13
12 Select13 the13 visiualizaon13 that13 you13 just13 created13
13
Page of 31 71 Page of 31 71
13
13 To13 save13 the13 dashboard13 click13 the13 Save13 Dashboard13 icon13
Page of 32 71 Page of 32 71
13
14 Add13 a13 tle13 and13 select13 whether13 to13 store13 the13 selected13 me13 frame13 with13 the13 dashboard13
13
15 Click13 Save13
16 You13 can13 edit13 the13 shape13 and13 number13 of13 charts13 and13 save13 the13 dashboard13
17 To13 re-shy‐run13 the13 dashboard13 select13 the13 Dashboard13 page13 and13 click13 the13 Load13 Saved13 Dashboard13 icon13
Page of 33 71 Page of 33 71
13
18 Select13 the13 saved13 dashboard13 from13 the13 list13
13
19 The13 dashboard13 runs13
13
Page of 34 71 Page of 34 71
Page of 35 71 Page of 35 71
Aler0ng13 with13 ELK13 ELK13 examines13 the13 log13 records13 in13 process13 to13 send13 alerts13 to13 third-shy‐party13 alerng13 event13 management13 and13 social13 media13 applicaons13 As13 shown13 previously13 in13 the13 ELK13 architecture13 the13 ELK13 strategy13 for13 alerng13 is13 to13 send13 alerts13 from13 the13 Logstash13 component13 directly13 Therefore13 log13 records13 and13 alerts13 are13 sent13 to13 both13 Elascsearch13 and13 third-shy‐party13 tools13 at13 the13 same13 instant13 using13 the13 output13 secon13 of13 the13 Logstash13 configuraon13 Any13 alerts13 are13 generated13 during13 the13 filter13 secon13 processing13 of13 the13 Logstash13 configuraon13
Create13 alerts13 in13 Logstash13 The13 Logstash13 filter13 secon13 is13 used13 to13 trigger13 alerts13 based13 on13 the13 log13 records13 being13 processed13 For13 example13 log13 records13 can13 be13 checked13 for13 errors13 warnings13 crashed13 or13 stopped13 messages13 and13 generate13 an13 alert13
Here13 is13 an13 example13 Logstash13 filter13 code13
13 if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
The13 specific13 tags13 are13 set13 against13 the13 log13 records13 to13 trigger13 the13 alert13 later13 in13 the13 configuraon13 in13 the13 output13 secon13
Control13 the13 number13 of13 generated13 alerts13 using13 the13 Logstash13 throcle13 plugin13 so13 that13 either13 a13 lot13 of13 instances13 in13 a13 fixed13 me13 period13 are13 required13 before13 an13 alert13 is13 sent13 or13 a13 specific13 number13 of13 instances13 triggers13 the13 alert13
Page of 36 71 Page of 36 71
Edit13 the13 plugin13 parameters13 to13 meet13 system13 requirements13
13 13 13 13 13 if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 10
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle13
after_count =gt 20
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 30
period =gt 1200
key =gt servicename_alarm_warning
Page of 37 71 Page of 37 71
add_tag =gt servicename_throttled
13 13 13 13 13 13 13 13 13
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 40
period =gt 1500
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
Enrich13 alerts13 with13 message13 detail13 Some13 of13 the13 third-shy‐party13 integraons13 benefit13 from13 having13 components13 of13 the13 raw13 log13 record13 sent13 in13 addion13 to13 the13 raw13 record13 or13 fields13 in13 the13 record13 edited13 transformed13 or13 translated13
Grok13 Logstash13 allows13 the13 log13 record13 to13 be13 broken13 up13 into13 logical13 secons13 and13 each13 secon13 stored13 in13 a13 new13 parameter13 using13 the13 grok13 parameter13 in13 the13 filter13 secon13 Here13 is13 an13 example13 grok13
13 grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
To13 build13 the13 grok13 statement13 Logstash13 offers13 an13 online13 tool13 at13 hcpsgrokdebugherokuappcom13
Translate13 If13 specific13 fields13 in13 the13 log13 record13 are13 coded13 (for13 example13 the13 Bluemix13 app13 name13 is13 oaen13 not13 sent13 in13 the13 syslog13 messages13 instead13 the13 GUID13 is13 sent13 which13 is13 not13 useful)13 then13 the13 filter13 secon13 translate13 plugin13 can13 be13 used13 to13 add13 new13 fields13 by13 holding13 the13 decoded13 versions13 of13 those13 values13
13 translate
field =gt data1
destination =gt data1
override =gt true
Page of 38 71 Page of 38 71
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
dc50f6e4-c052-436a-bf27-95ba392c4228 monverifyservicemanagement
0c677e82-5bfc-4e38-9155-e416249e3ee6 ms-catalog-postpyretic-pomiculture
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
However13 the13 translate13 plugin13 is13 not13 included13 with13 the13 Logstash13 installaon13 In13 the13 next13 secon13 I13 will13 13 describe13 how13 to13 add13 new13 plugins13 to13 the13 Logstash13 instance13
Add13 custom13 plugins13 to13 Logstash13 As13 an13 open13 source13 tool13 Logstash13 encourages13 users13 to13 develop13 content13 for13 all13 users13 to13 share13 Some13 of13 the13 plugins13 that13 were13 created13 by13 our13 community13 include13 a13 translaon13 plugin13 an13 output13 Slack13 plugin13 and13 an13 output13 syslog13 plugin13 Letrsquos13 look13 at13 how13 to13 add13 these13 plugins13 to13 your13 environment13
Filter13 translate13 plugin13 To13 add13 the13 translate13 plugin13
1 Download13 the13 plugin13
2 Copy13 the13 plugin13 file13 logstash-shy‐filter-shy‐translate-shy‐214gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-filter-translate 214
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Output13 Slack13 plugin13 To13 add13 the13 Slack13 plugin13
Page of 39 71 Page of 39 71
1 Download13 the13 plugin13 13
2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐slack-shy‐014gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-output-slack 014
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Output13 syslog13 plugin13 To13 add13 the13 syslog13 plugin13
1 Download13 the13 plugin13 13
2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐syslog-shy‐214gem13 to13 the13 ELK13 Virtual13 Server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-output-syslog 214
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Integrate13 Logstash13 with13 Slack13 To13 send13 generated13 alerts13 to13 Slack13 complete13 these13 steps13
1 Create13 a13 Slack13 integraon13 13
2 Use13 the13 Slack13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 Slack13 integraon13
113 Login13 to13 Slack13 and13 go13 to13 the13 Apps13 page13 13
Page of 40 71 Page of 40 71
13
313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13
13
A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13
13
Page of 42 71 Page of 42 71
The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13
13
The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13
13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
slack
url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5
channel =gt cloud-operations-mon
username =gt nick_cawood
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13
Page of 43 71 Page of 43 71
Page of 44 71 Page of 44 71
In13 Slack13 the13 alerts13 look13 like13 this13
13
Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13
1 Create13 a13 PagerDuty13 integraon13 13
2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13
1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13
13
2 Click13 Add13 New13 Service13
13
313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13
Page of 45 71 Page of 45 71
13
413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13
513 Click13 Add13 Service13
13
The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13
Page of 46 71 Page of 46 71
13
The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt be7eb1ea9c224018923e818d3e891d0b
incident_key =gt logstashhosttype
Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13
Page of 47 71 Page of 47 71
13
Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13
1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13
2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13
3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13
4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13
13 13 http
url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo
http_method =gt post
format =gt json
For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13
13 13 syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
Page of 48 71 Page of 48 71
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13
13
The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13
if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2
This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13
Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13
Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13
Logstash13 severity Shown13 in13 Omnibus13 as
alert Indeterminate
crical Warning
debug 7
emergency Clear
error Minor
informaonal 6
noce Crical
warning Major
Page of 49 71 Page of 49 71
13
Page of 50 71 Page of 50 71
Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13
input
tcp
port =gt 5000
type =gt syslog
filter
mutate
add_field =gt format =gt cf
grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
translate
field =gt data1
destination =gt data1
override =gt true
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
=== Check Alarm Criteria ===
Standard Key Words
if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
=== Alarm Exclusions ===
if [message] =~ (i)sometexthere
mutate remove_tag =gt servicename_alarm
Throttle alarms so we do not hit PD rate limits
Page of 52 71 Page of 52 71
if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 5
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle
after_count =gt 5
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 5
period =gt 1200
key =gt servicename_alarm_warning
add_tag =gt servicename_throttled
Page of 53 71 Page of 53 71
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
output
elasticsearch
hosts =gt localhost
stdout codec =gt rubydebug
if servicename_alarm_err in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
Page of 54 71 Page of 54 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 55 71 Page of 55 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
Page of 56 71 Page of 56 71
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert err message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_error in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
Page of 57 71 Page of 57 71
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 58 71 Page of 58 71
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 59 71 Page of 59 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert error message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
Page of 60 71 Page of 60 71
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
Page of 61 71 Page of 61 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 62 71 Page of 62 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt x
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert fatal message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
Page of 63 71 Page of 63 71
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
Page of 64 71 Page of 64 71
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 65 71 Page of 65 71
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt heavy_exclamation_mark
Page of 66 71 Page of 66 71
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert WARNING message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 67 71 Page of 67 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
Page of 68 71 Page of 68 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 69 71 Page of 69 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
Page of 70 71 Page of 70 71
Page of 71 71 Page of 71 71
CONTENTS13
Introducon13 313
What13 is13 Elascsearch13 Logstash13 Kibana13 313
LOGSTASH 3 ELASTICSEARCH 4 KIBANA 4 ELK SYSTEM ARCHITECTURE 4
Service13 Management13 architecture13 613
Log13 Monitoring13 System13 architecture13 613
Bluemix13 environment13 system13 architecture13 613
Creang13 a13 virtual13 server13 in13 Bluemix13 713
CREATE AN SSH KEY 8 PROVISIONING A VIRTUAL SERVER 9 SET UP THE VIRTUAL SERVER 12
Log on to the virtual server 12 Acces the root user 12 Copy files to the virtual server 13 Install and access the virtual server via VNC server 13
Install13 the13 ELK13 stack13 1313
INSTALL ELASTICSEARCH 14 Start Elasticsearch 14
INSTALL KIBANA 15 Start Kibana 18
BROADCAST BLUEMIX APP LOG RECORDS 19 INSTALL LOGSTASH 19
Write Logstash configuration 19 Start Logstash 21 Monitor Logstash processing 21
Use13 ELK13 2213
KIBANA DISCOVER PAGE 22 Change the time filter 22 Custom search queries 23 Saved searches 24
BUILD SIMPLE KIBANA HISTOGRAMS 27 Alerng13 with13 ELK13 3613
CREATE ALERTS IN LOGSTASH 36 ENRICH ALERTS WITH MESSAGE DETAIL 38
Grok 38 Translate 38
ADD CUSTOM PLUGINS TO LOGSTASH 39 Filter translate plugin 39 Output Slack plugin 39 Output syslog plugin 40
INTEGRATE LOGSTASH WITH SLACK 40 INTEGRATE LOGSTASH WITH PAGERDUTY 45 INTEGRATE LOGSTASH WITH IBM NETCOOL OMNIBUS 48
Appendix13 Logstash13 configuraon13 51
Page of 2 71
Introduc0on13 IBMreg13 Bluemixreg13 IBMrsquos13 PlaRorm13 as13 a13 Service13 (PaaS)13 allows13 you13 to13 rapidly13 build13 and13 deploy13 an13 applicaon13 Applicaons13 can13 run13 in13 a13 nave13 or13 a13 hybrid13 Bluemix13 environment13 13 13
This13 document13 describes13 the13 integraon13 of13 the13 Elascsearch13 Logstash13 Kibana13 (ELK)13 log13 analycs13 open13 source13 stack13 into13 the13 Bluemix13 environment13 how13 to13 collect13 logs13 from13 Bluemix13 applicaons13 (apps)13 and13 send13 alerts13 from13 those13 log13 records13 to13 selected13 alertevent13 noficaon13 targets13 (PagerDuty13 Slack13 and13 IBM13 Netcool13 Omnibus)13
This13 document13 describes13 how13 to13 provision13 Bluemix13 virtual13 servers13 in13 the13 Bluemix13 environment13 install13 the13 ELK13 stack13 on13 a13 Bluemix13 virtual13 server13 and13 configure13 the13 ELK13 stack13 to13 integrate13 with13 other13 components13 deployed13 on13 the13 Bluemix13 environment13
The13 steps13 to13 integrate13 the13 ELK13 stack13 into13 the13 Cloud13 Architecture13 and13 Soluon13 Engineering13 (CASE)13 environment13 are13 used13 as13 the13 basis13 for13 this13 document13 13 For13 reference13 both13 the13 ELK13 architecture13 within13 the13 CASE13 environment13 and13 the13 whole13 CASE13 environment13 are13 defined13 using13 system13 architecture13 diagrams13 as13 well13 as13 diagrams13 defining13 the13 ELK13 structure13 and13 ELK13 integraon13 into13 Bluemix13 architecture13
What13 is13 Elas0csearch13 Logstash13 Kibana13 Elascsearch13 Logstash13 Kibana13 is13 an13 open13 source13 stack13 of13 three13 applicaons13 that13 work13 together13 to13 provide13 an13 end-shy‐to-shy‐end13 search13 and13 visualisaon13 plaRorm13 for13 the13 invesgaon13 of13 log13 file13 sources13 in13 real13 me13 Log13 file13 records13 can13 be13 searched13 from13 a13 variety13 of13 log13 sources13 charts13 and13 dashboards13 built13 to13 visualize13 the13 log13 records13 The13 three13 applicaons13 are13 Elascsearch13 Logstash13 and13 Kibana13
Logstash13 Logstash13 is13 an13 applicaon13 that13 allows13 the13 ELK13 stack13 to13 manage13 the13 gathering13 transformaon13 and13 transport13 of13 log13 file13 records13 Logstash13 is13 able13 to13 pull13 records13 from13 remote13 sources13 or13 listen13 for13 records13 on13 specified13 ports13
Logstash13 has13 three13 areas13 of13 funcon13
bull Process13 input13 of13 the13 log13 file13 records13 from13 the13 remote13 source13 You13 can13 configure13 Logstash13 to13 receive13 records13 in13 a13 variety13 of13 methods13 for13 example13 reading13 from13 a13 flat13 file13 or13 from13 a13 TCP13 port13 or13 direct13 from13 a13 remote13 or13 local13 syslog13 These13 methods13 are13 defined13 in13 the13 Logstash13 online13 support13 and13 input13 secon13 of13 the13 Logstash13 conf13 file13 Once13 all13 input13 processing13 is13 completed13 Logstash13 moves13 onto13 the13 next13 funcon13
bull Transform13 and13 enrich13 the13 log13 records13 that13 take13 place13 in13 the13 Logstash13 filter13 secon13 Any13 number13 of13 changes13 to13 the13 format13 (and13 content13 if13 required)13 of13 the13 log13 file13 record13 can13 be13 made13 using13 the13 various13 methods13 provided13 with13 Logstash13 for13 example13 methods13 to13 add13 and13 enrich13 log13 record13 fields13 and13 the13 flagging13 I13 describe13 excluding13 and13 liming13 records13 for13 alerng13 later13 in13 this13 document13 Aaer13 the13 log13 record13 is13 transformed13 Logstash13 processes13 the13 record13
bull Send13 the13 log13 records13 to13 the13 Elascsearch13 applicaon13 from13 the13 output13 secon13 of13 the13 Logstash13 configuraon13 A13 specific13 elascsearch13 method13 is13 included13 with13 the13 Logstash13 instance13 that13 sends13 the13 log13 records13 to13 the13 Elascsearch13 instance13 However13 as13 with13 the13 input13 and13 filter13
Page of 3 71 Page of 3 71
secons13 a13 variety13 of13 methods13 are13 available13 to13 process13 the13 data13 sent13 from13 the13 filter13 secon13 For13 example13 the13 records13 can13 be13 sent13 to13 external13 alerng13 tools13 such13 as13 PagerDuty13 social13 media13 tools13 such13 as13 Slack13 or13 event13 management13 hubs13 like13 IBM13 Netcool13 Omnibus13 via13 output13 methods13 syslog13 or13 hcp13 13
As13 an13 open13 source13 tool13 non-shy‐Logstash13 employees13 develop13 other13 methods13 and13 share13 them13 with13 the13 user13 community13 for13 addion13 to13 the13 library13 of13 methods13 Logstash13 can13 execute13 (examples13 of13 this13 process13 are13 included13 later13 in13 this13 document)13
Elas0csearch13 Elascsearch13 is13 the13 index13 store13 and13 query13 applicaon13 on13 the13 ELK13 stack13 It13 provides13 a13 standard13 full13 text13 search13 facility13 of13 the13 log13 file13 records13 processed13 by13 Logstash13 The13 ELK13 online13 help13 documentaon13 describes13 Elascsearch13 as13 follows13 ldquoIt13 is13 generally13 used13 as13 the13 underlying13 enginetechnology13 that13 powers13 applicaons13 that13 have13 complex13 search13 features13 and13 requirementsrdquo13
Kibana13 Kibana13 is13 the13 user13 interface13 for13 the13 ELK13 stack13 Kibana13 integrates13 with13 Elascsearch13 to13 visualise13 the13 results13 of13 searches13 and13 allow13 the13 creaon13 of13 charts13 and13 dashboards13 based13 on13 the13 searches13 The13 searches13 charts13 and13 dashboard13 can13 visualise13 real-shy‐me13 or13 historical13 log13 records13 Users13 access13 Kibana13 through13 an13 hcp13 URL13 from13 a13 client13 web13 browser13
ELK13 System13 Architecture13 The13 ELK13 stack13 can13 be13 described13 as13 follows13
Page of 4 71 Page of 4 71
13
Page of 5 71 Page of 5 71
Service13 Management13 architecture13 The13 IBM13 Cloud13 Architecture13 Center13 created13 a13 standard13 reference13 architecture13 for13 how13 to13 set13 up13 your13 cloud13 environemnt13 to13 handle13 incident13 management13 13
View13 the13 standard13 Service13 Management13 architecure13
13 13
Log13 Monitoring13 System13 architecture13 The13 following13 diagram13 shows13 a13 standard13 architecture13 for13 the13 log-shy‐monitoring13 system13 described13 in13 this13 tutorial13 13
13
Bluemix13 environment13 system13 architecture13 The13 ELK13 stack13 can13 be13 installed13 into13 a13 variety13 of13 environments13 When13 using13 Bluemix13 the13 following13 figure13 shows13 what13 the13 environment13 might13 look13 like13
Page of 6 71 Page of 6 71
13
The13 Bluemix13 cloud13 plaRorm13 offers13 a13 Public13 network13 and13 access13 to13 US13 South13 and13 UK13 environments13 From13 all13 these13 environments13 third-shy‐party13 cloud-shy‐based13 soluons13 are13 also13 accessible13
The13 ELK13 stack13 is13 installed13 in13 the13 Bluemix13 United13 Kingdom13 environment13 because13 this13 is13 the13 environment13 where13 Bluemix13 has13 virtual13 servers13 available13 for13 provisioning13 Other13 applicaons13 in13 the13 CASE13 project13 were13 also13 installed13 on13 virtual13 servers13 in13 the13 Bluemix13 United13 Kingdom13 region13
Bluemix13 apps13 were13 used13 to13 create13 a13 system13 to13 be13 monitored13 by13 ELK13 These13 apps13 were13 hosted13 in13 the13 Bluemix13 US13 South13 environment13 using13 Bluemix13 microservices13 which13 produce13 logs13 during13 normal13 operaon13 and13 during13 crashes13 and13 outages13
Through13 the13 Bluemix13 Cloud13 Foundry13 process13 the13 log13 records13 can13 be13 broadcast13 from13 the13 Bluemix13 apps13 so13 Logstash13 can13 receive13 them13 and13 send13 them13 to13 Elascsearch13 within13 the13 ELK13 stack13 or13 other13 virtual13 servers13 (hosng13 other13 IBM13 or13 third-shy‐party13 products)13 in13 the13 Bluemix13 United13 Kingdom13 environment13 or13 to13 third-shy‐party13 applicaons13 in13 the13 public13 cloud13 (Pager13 Duty13 Slack13 etc)13
Crea0ng13 a13 virtual13 server13 in13 Bluemix13 With13 Bluemix13 you13 can13 provision13 a13 virtual13 server13 in13 the13 cloud13 to13 install13 standalone13 or13 integrated13 system13 components13 Not13 all13 Bluemix13 regions13 offer13 this13 service13 all13 the13 me13 due13 to13 beta-shy‐project13 constraints13 In13 this13 arclersquos13 examples13 virtual13 servers13 were13 allowed13 to13 be13 provisioned13 in13 the13 United13 Kingdom13 region13 For13 more13 informaon13 read13 how13 to13 provision13 a13 Bluemix13 virtual13 server13 including13 some13 simple13 setup13 steps13 for13 ease13 of13 use13
Page of 7 71 Page of 7 71
Create13 an13 SSH13 key13 To13 access13 your13 virtual13 server13 once13 it13 is13 provisioned13 allocate13 an13 SSH13 key13 to13 the13 virtual13 server13
To13 generate13 a13 publicprivate13 RSA13 key13 pair13 follow13 these13 steps13
1 In13 a13 UNIX13 or13 Linuxreg13 window13 run13 the13 following13 command13 specifying13 a13 ltuniquekeynamegt13 of13 your13 your13 choice13 and13 a13 password13 or13 passphrase13 (memorize13 the13 password)$13 ssh-shy‐keygen13 -shy‐t13 rsa13 -shy‐f13 ltuniquekeynamegt13
2 Enter13 the13 passphrase13 (empty13 for13 no13 passphrase)13
3 Enter13 the13 same13 passphrase13 again13
Your13 idenficaon13 has13 been13 saved13 in13 ltuniquekeynamegt13
Your13 public13 key13 has13 been13 saved13 in13 ltuniquekeynamegtpub13
The13 key13 fingerprint13 is13
d6268464506525930ce4632c693c203c13 ltusergtltservergt13
The13 keys13 randomart13 image13 is13
+--[ RSA 2048]----+
|o ++=o |
| E o ooo |
| |
| + o |
| S o |
| o |
| |
| |
| |
+-----------------+
Two13 keys13 are13 created13 public13 and13 private13 The13 public13 key13 contents13 are13 used13 to13 create13 a13 virtual13 server13 with13 a13 public13 IP13 address13 The13 private13 key13 is13 used13 to13 access13 the13 virtual13 server13 from13 a13 client13 machine13
Page of 8 71 Page of 8 71
Provisioning13 a13 virtual13 server13 1 A13 Bluemix13 account13 is13 required13 for13 this13 process13 Sign13 up13 here13 and13 log13 in13 Make13 sure13 your13
environment13 is13 correct13 For13 example13 if13 you13 are13 in13 the13 United13 Kingdom13 select13 the13 United13 Kingdom13 region13 in13 the13 Bluemix13 console13
13
13
2 Add13 a13 virtual13 server13 to13 your13 Bluemix13 environment13 Click13 Run13 Virtual13 Servers13
13
3 Select13 an13 OS13 image13 from13 the13 EXISTING13 list13 For13 ELK13 select13 Centos13 OS13 (use13 latest13 available13 version)13
Page of 9 71 Page of 9 71
4 Name13 the13 virtual13 server13
5 Make13 sure13 Assign13 public13 IP13 address13 is13 checked13
13
6 Click13 Add13 Key13 to13 add13 the13 SSH13 key13 you13 created13 previously13 Copy13 the13 contents13 of13 the13 public13 key13 into13 the13 Public13 Key13 to13 import13 field13
Page of 10 71 Page of 10 71
13
7 Click13 OK13 Then13 click13 CREATE13
The13 virtual13 server13 is13 provisioned13 and13 added13 to13 the13 Bluemix13 dashboard13 under13 VIRTUAL13 SERVERS13
Page of 11 71 Page of 11 71
13
Set13 up13 the13 virtual13 server13
Log13 on13 to13 the13 virtual13 server13 Now13 you13 can13 log13 on13 to13 the13 virtual13 server13 and13 add13 new13 users13 and13 programs13 to13 make13 it13 easier13 to13 access13 the13 virtual13 server13
To13 access13 the13 server13 go13 to13 the13 UNIX13 or13 Linux13 window13 that13 you13 used13 to13 create13 the13 SSH13 key13 In13 the13 directory13 where13 the13 ltuniquekeynamegt13 files13 exist13 run13 the13 following13 command13 referencing13 the13 SSH13 keyname13 the13 public13 IP13 address13 Bluemix13 allocated13 for13 the13 virtual13 server13 and13 the13 password13 for13 the13 SSH13 key13
[userserver ~]$ ssh -i ltuniquekeynamegt ibmcloudltpublicIPaddressgt
Enter13 the13 passphrase13 for13 the13 key13 ltuniquekeynamegt13
Last login Tue Jul 5 125546 2016 from 108-210-96-231lightspeedrlghncsbcglobalnet
[ibmcloudelkforcase ~]$
ibmcloud13 is13 the13 default13 user13 for13 the13 Bluemix13 virtual13 servers13 and13 allows13 SSH13 to13 access13 the13 server13
Acces13 the13 root13 user13 To13 access13 the13 root13 user13 log13 on13 as13 ibmcloud13 user13 and13 run13 the13 following13 command13
[ibmcloudelkforcase ~]$ sudo bash
[rootelkforcase ibmcloud]
The13 root13 user13 can13 be13 used13 to13 add13 programs13 to13 the13 server13 and13 create13 and13 edit13 users13 as13 required13 to13 access13 applicaons13 installed13 on13 the13 virtual13 server13
Page of 12 71 Page of 12 71
Copy13 files13 to13 the13 virtual13 server13 To13 move13 installaon13 files13 to13 the13 server13 go13 to13 the13 UNIX13 or13 Linux13 window13 you13 used13 to13 create13 the13 SSH13 key13 and13 the13 directory13 where13 the13 ltuniquekeynamegt13 files13 exist13 Move13 the13 installaon13 files13 to13 the13 same13 locaon13
Run13 the13 following13 command13 referencing13 the13 SSH13 keyname13 the13 public13 IP13 address13 Bluemix13 allocated13 for13 the13 virtual13 server13 and13 the13 password13 for13 the13 SSH13 key13
[userserver ~]$ scp ndashi ltuniquekeynamegt ltINSTALLATIONFILENAMEgt ibmcloudltpublicIPaddressgthomeibmcloud
Enter13 the13 passphrase13 for13 key13 cloudkey313
This13 copies13 the13 file13 using13 the13 ibmcloud13 user13 to13 the13 homeibmcloud13 directory13 on13 the13 virtual13 server13
Install13 and13 access13 the13 virtual13 server13 via13 VNC13 server13 Using13 the13 ssh13 command13 to13 access13 the13 virtual13 server13 is13 not13 the13 most13 stable13 method13 and13 does13 not13 offer13 X-shy‐Windows13 support13 Therefore13 we13 suggest13 that13 you13 use13 a13 VNC13 client13 to13 access13 the13 virtual13 server13
Installing13 a13 VNC13 client13 should13 be13 done13 as13 the13 root13 user13 Use13 the13 yum13 command13 as13 follows13
[ibmcloudelkforcase ~]$ yum groupinstall GNOME Desktop
[ibmcloudelkforcase ~]$ yum install tigervnc-server
These13 commands13 first13 install13 a13 desktop13 style13 OS13 user13 interface13 and13 then13 a13 VNC13 client13 to13 remotely13 access13 the13 desktop13 style13 OS13
Then13 as13 either13 the13 root13 user13 or13 another13 user13 (example13 below13 uses13 elk)13 run13 the13 following13 command13 to13 create13 the13 VNC13 client13 connecon13
[elkelkforcase ~]$ vncserver
New elkforcase2 (elk) desktop is elkforcase2
Starting applications specified in homeelkvncxstartup
Log file is homeelkvncelkforcase2log
[elkelkforcase ~]$
Install13 a13 VNC13 client13 on13 the13 client13 machine13 used13 to13 access13 the13 Internet13 and13 (in13 the13 example13 above)13 use13 the13 command13 ltpublicIPadddressgt213 and13 the13 password13 for13 ELK13 to13 access13 the13 virtual13 server13
Using13 a13 VNC13 client13 means13 that13 you13 can13 leave13 long13 duraon13 installaons13 running13 without13 fear13 of13 the13 terminal13 losing13 connecon13 and13 the13 installaons13 failing13
Install13 the13 ELK13 stack13 Download13 the13 source13 soaware13 code13 for13 the13 ELK13 stack13 from13 the13 ELK13 website13 13
At13 the13 me13 of13 wring13 the13 downloaded13 soaware13 files13 were13
Page of 13 71 Page of 13 71
13
Transfer13 the13 files13 to13 the13 virtual13 server13 (using13 the13 scp13 method13 detailed13 above)13
Install13 Elas0csearch13 As13 a13 root13 user13 create13 a13 non-shy‐root13 user13 with13 the13 adduser13 command13
Create13 a13 directory13 to13 install13 Elascsearch13 into13 and13 untar13 the13 contents13 of13 the13 Elascsearch13 install13 file13 into13 that13 directory13
[elkelkforcase]$ cd home
[elkelkforcase]$ mkdir elasticsearch
[elkelkforcase]$ cp homeibmcloudelasticsearch-233targz homeelasticsearch
[elkelkforcase]$ cd elasticsearch
[elkelkforcase]$ gunzip elasticsearch-233targz
[elkelkforcase]$ untar elasticsearch-233tar
Elascsearch13 is13 now13 installed13
Start13 Elas0csearch13 To13 start13 Elascsearch13 go13 to13 the13 bin13 directory13 as13 the13 non-shy‐root13 user13 when13 you13 installed13 it13 and13 run13 the13 elascsearch13 executable13
[elkelkforcase]$ cd homeelasticsearchelasticsearch-233bin
[elkelkforcase]$ elasticsearch amp
[2] 22211
[elkelkforcase]$ [2016-07-05 133857730][INFO ][node] [Arsenal] version[233] pid[22211] build[218bdf12016-05-17T154004Z]
[2016-07-05 133857740][INFO ][node] [Arsenal] initializing
[2016-07-05 133858984][INFO ][plugins] [Arsenal] modules [reindex lang-expression lang-groovy] plugins [] sites []
[2016-07-05 133859146][INFO ][env] [Arsenal] using [1] data paths mounts [[ (rootfs)]] net usable_space [155gb] net total_space [199gb] spins [unknown] types [rootfs]
[2016-07-05 133859146][INFO ][env] [Arsenal] heap size [10156mb] compressed ordinary object pointers [true]
Page of 14 71 Page of 14 71
[2016-07-05 133859147][WARN ][env] [Arsenal] max file descriptors [4096] for elasticsearch process likely too low consider increasing to at least [65536]
[2016-07-05 133903526][INFO ][node] [Arsenal] initialized
[2016-07-05 133903526][INFO ][node] [Arsenal] starting
To13 test13 that13 Elascsearch13 is13 running13 correctly13 open13 the13 browser13 in13 the13 VNC13 client13 and13 run13 localhost920013
13
Install13 Kibana13 ELK13 recommends13 installing13 and13 running13 Kibana13 as13 root13 13
1 As13 root13 user13 create13 an13 installaon13 directory13 and13 untar13 the13 installaon13 file13
[rootelkforcase]$ cd home
[rootelkforcase]$ mkdir kibana
[rootelkforcase]$ cp homeibmcloudkibana-451-linux-x64targz homekibana
[rootelkforcase]$ cd kibana
[rootelkforcase]$ gunzip kibana-451-linux-x64targz
[rootelkforcase]$ untar kibana-451-linux-x64tar
2 Modify13 the13 manifestyml13 configuraon13 file13 Go13 to13 the13 kibana-shy‐451-shy‐linux-shy‐x6413 directory13
[rootelkforcase]$ cd homekibanakibana-451-linux-x64
3 Edit13 the13 manifestyml13 file13 so13 the13 contents13 are13
applicaons13
-shy‐ name13 kibana-shy‐in-shy‐bluemix13
memory13 128M13
host13 kibana-shy‐in-shy‐bluemix13
buildpack13 hcpsgithubcomcloudfoundry-shy‐communitynginx-shy‐buildpackgit13
Page of 15 71 Page of 15 71
4 Edit13 the13 configkibanayml13 file13 so13 the13 contents13 are13
Kibana is served by a back end server This controls which port to use
port 5601
The host to bind the server to
host 0000
If you are running kibana behind a proxy and want to mount it at a path
specify that path here The basePath cant end in a slash
serverbasePath
The maximum payload size in bytes on incoming server requests
servermaxPayloadBytes 1048576
The Elasticsearch instance to use for all your queries
elasticsearchurl httplocalhost9200
preserve_elasticsearch_host true will send the hostname specified in `elasticsearch` If you set it to false
then the host you use to connect to this Kibana instance will be sent
elasticsearchpreserveHost true
Kibana uses an index in Elasticsearch to store saved searches visualizations
and dashboards It will create a new index if it doesnt already exist
kibanaindex kibana
The default application to load
kibanadefaultAppId discover
If your Elasticsearch is protected with basic auth these are the user credentials
used by the Kibana server to perform maintenance on the kibana_index at startup Your
Kibana users will still need to authenticate with Elasticsearch (which is proxied through
the Kibana server)
Page of 16 71 Page of 16 71
elasticsearchusername user
elasticsearchpassword pass
SSL for outgoing requests from the Kibana Server to the browser (PEM formatted)
serversslcert pathtoyourservercrt
serversslkey pathtoyourserverkey
Optional setting to validate that your Elasticsearch backend uses the same key files (PEM formatted)
elasticsearchsslcert pathtoyourclientcrt
elasticsearchsslkey pathtoyourclientkey
If you need to provide a CA certificate for your Elasticsearch instance put
the path of the pem file here
elasticsearchsslca pathtoyourCApem
Set to false to have a complete disregard for the validity of the SSL
certificate
elasticsearchsslverify true
Time in milliseconds to wait for elasticsearch to respond to pings defaults to
request_timeout setting
elasticsearchpingTimeout 1500
Time in milliseconds to wait for responses from the back end or elasticsearch
This must be gt 0
elasticsearchrequestTimeout 30000
Time in milliseconds for Elasticsearch to wait for responses from shards
Set to 0 to disable
elasticsearchshardTimeout 0
Time in milliseconds to wait for Elasticsearch at Kibana startup before retrying
elasticsearchstartupTimeout 5000
elasticsearchstartupTimeout 30000
Page of 17 71 Page of 17 71
Set the path to where you would like the process id file to be created
pidfile varrunkibanapid
If you would like to send the log output to a file you can set the path below
loggingdest stdout
Set this to true to suppress all logging output
loggingsilent false
Set this to true to suppress all logging output except for error messages
loggingquiet false
Set this to true to log all events including system usage information and all requests
loggingverbose false
Kibana13 is13 now13 installed13
Start13 Kibana13 To13 start13 Kibana13 go13 to13 the13 bin13 directory13 and13 run13 the13 kibana13 executable13
[rootelkforcase]$
[rootelkforcase]$ cd homekibanakibana-451-linux-x64bin
[rootelkforcase]$ kibana amp
[1] 22283
[rootelkforcase] log [095900542] [info][status][pluginkibana] Status changed from uninitialized to green - Ready
log [095900600] [info][status][pluginelasticsearch] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [095900689] [info][status][pluginkbn_vislib_vis_types] Status changed from uninitialized to green - Ready
log [095900717] [info][status][pluginmarkdown_vis] Status changed from uninitialized to green - Ready
log [095900721] [info][status][pluginmetric_vis] Status changed from uninitialized to green - Ready
log [095900814] [info][status][pluginspyModes] Status changed from uninitialized to green - Ready
log [095900934] [info][status][pluginstatusPage] Status changed from uninitialized to green - Ready
Page of 18 71 Page of 18 71
log [095900941] [info][status][plugintable_vis] Status changed from uninitialized to green - Ready
log [095901067] [info][listening] Server running at http00005601
log [095906083] [info][status][pluginelasticsearch] Status changed from yellow to yellow - No existing Kibana index found
log [095908942] [info][status][pluginelasticsearch] Status changed from yellow to green - Kibana index ready
Kibana13 is13 now13 started13
Broadcast13 Bluemix13 app13 log13 records13 Bluemix13 apps13 (which13 are13 shown13 in13 the13 US13 South13 Bluemix13 region13 of13 the13 System13 Architecture13 diagram)13 will13 produce13 logs13 in13 a13 syslog13 rfc542413 format13 using13 a13 Bluemix13 tool13 called13 Loggregator13
For13 Logstash13 to13 receive13 these13 logs13 each13 Bluemix13 app13 needs13 to13 broadcast13 these13 logs13 To13 do13 that13 run13 the13 following13 commands13 on13 each13 Bluemix13 app13 using13 the13 public13 IP13 address13 of13 the13 ELK13 virtual13 server13 the13 ELK13 tag13 to13 trace13 the13 process13 and13 insert13 the13 name13 of13 each13 app13 in13 turn13
cf cups elk -l syslogltpublicIPaddress5000 cf bind-service ltmyappgt elk
cf restage ltmyappgt
Install13 Logstash13 To13 install13 Logstash13 as13 root13 user13 create13 a13 directory13 for13 the13 installaon13 and13 untar13 the13 installaon13 file13 there13
13 13 [rootelkforcase]$ cd home
[rootelkforcase]$ mkdir logstash
[rootelkforcase]$ cp homeibmcloudlogstash-233targz homelogstash
[rootelkforcase]$ cd logstash
[rootelkforcase]$ gunzip logstash-233targz
[rootelkforcase]$ untar logstash-233tar
Write13 Logstash13 configura0on13 Logstash13 requires13 a13 configuraon13 file13 to13 be13 wricen13 to13 receive13 transform13 and13 output13 the13 log13 records13 to13 Elascsearch13 This13 secon13 explains13 the13 basic13 requirements13 of13 processing13 Bluemix13 app13 log13 records13 into13 Elascsearch13 (more13 complex13 configuraons13 are13 detailed13 later13 in13 this13 document13 for13 the13 integraons13 with13 third-shy‐party13 tools)13
Create13 a13 conf13 file13 as13 root13 user13
[rootelkforcase]$ cd homelogstashlogstash-233
[rootelkforcase]$ mkdir config
Page of 19 71 Page of 19 71
[rootelkforcase]$ cd config
[rootelkforcase]$ touch loggregatorconf
Edit13 the13 loggregator13 file13 with13 the13 following13 contents13
13 Input13 secon13 defines13 which13 log13 records13 to13 receive13
13 port13 and13 type13 reflect13 method13 selected13 for13 Cloud13 Foundry13 to13 broadcast13 the13 records13
input
tcp
port =gt 5000
type =gt syslog
13 Filter13 secon13 ndash13 transforms13 the13 data13
filter
Add a field to identify the records as Cloud Foundry based
mutate
add_field =gt format =gt cf
13 Output13 secon13 ndash13 sends13 the13 data13 to13 its13 desnaon(s)13
output
13 Use13 the13 elascsearch13 plugin13 to13 send13 the13 data13 to13 the13 elascsearch13 instance13
13 elasticsearch
hosts =gt localhost
stdout codec =gt rubydebug
13 End13 of13 Config13
Save13 the13 file13 Verify13 the13 format13 of13 the13 file13 is13 acceptable13 by13 running13 the13 following13 commands13
[rootelkforcase]$ cd homelogstashlogstash-233
[rootelkforcase]$ binlogstash agent --configtest -f configloggregatorconf
Page of 20 71 Page of 20 71
Configuraon13 is13 OK13
Start13 Logstash13 Aaer13 you13 have13 wricen13 and13 verified13 the13 Logstash13 configuraon13 you13 can13 start13 Logstash13 to13 process13 records13 from13 the13 Bluemix13 apps13 (via13 Cloud13 Foundry)13 Run13 the13 following13 as13 root13 user13
[rootelkforcase]$ cd homelogstashlogstash-233
[rootelkforcase]$ binlogstash agent -f configloggregatorconf
Logstash13 is13 now13 started13
Monitor13 Logstash13 processing13 Once13 Logstash13 is13 running13 and13 log13 records13 are13 being13 produced13 by13 the13 Bluemix13 apps13 you13 can13 monitor13 progress13 in13 the13 same13 window13 that13 was13 used13 to13 start13 Logstash13 Output13 is13 similar13 to13 the13 following13
13
The13 log13 records13 being13 broadcast13 from13 the13 Bluemix13 apps13 are13 now13 sent13 to13 Elascsearch13 from13 the13 Logstash13 output13 configuraon13 Records13 are13 sent13 in13 real13 me13 If13 Logstash13 is13 not13 running13 while13 log13 records13 are13 being13 produced13 then13 these13 records13 will13 not13 be13 processed13 by13 Logstash13 and13 will13 not13 be13 stored13 in13 Elascsearch13
Page of 21 71 Page of 21 71
Use13 ELK13 The13 log13 records13 from13 the13 Bluemix13 apps13 are13 now13 being13 stored13 and13 indexed13 by13 the13 Elascsearch13 component13 This13 data13 can13 be13 accessed13 via13 the13 Kibana13 applicaon13
Kibana13 Discover13 page13 Using13 a13 web13 browser13 either13 directly13 from13 a13 client13 or13 using13 the13 VNC13 client13 access13 Kibana13 using13 the13 publicIPaddress13 of13 the13 ELK13 virtual13 server13
hcpltpublicIPaddressgt560113
No13 userpassword13 credenals13 are13 required13
Kibana13 defaults13 to13 a13 search13 of13 all13 records13 ()13 in13 the13 last13 1513 minutes13 and13 returns13 the13 results13
13
More13 complex13 searches13 are13 possible13 by13 changing13 the13 me13 filter13 and13 supplying13 search13 queries13
Change13 the13 0me13 filter13 To13 change13 the13 me13 ranges13 click13 the13 clock13 symbol13 in13 the13 top-shy‐right13 of13 the13 Kibana13 Discover13 page13
13
You13 will13 see13 the13 following13 opons13
13
Page of 22 71 Page of 22 71
A13 variety13 of13 quick13 me13 ranges13 can13 be13 selected13 or13 you13 can13 switch13 to13 relave13 or13 absolute13 me13 ranges13
Quick13 me13 ranges13 allow13 switching13 between13 last13 X13 minutes13 to13 last13 Y13 hours13 or13 Z13 days13 etc13
Relave13 me13 ranges13 allow13 more13 specific13 versions13 of13 the13 Quick13 opons13 from13 specific13 minuteshoursdays13 to13 now13
13
Absolute13 me13 ranges13 specify13 the13 start13 and13 end13 mes13 for13 the13 searches13
13
Custom13 search13 queries13 You13 can13 use13 the13 text13 window13 of13 the13 Discover13 page13 to13 search13 for13 matches13 within13 the13 log13 records13 You13 can13 use13 Apache13 Lucene13 or13 Elascsearch13 query13 DSL13 syntax13
13
Page of 23 71 Page of 23 71
Saved13 searches13 You13 can13 save13 any13 custom13 search13 syntax13 and13 choice13 of13 me13 range13 for13 future13 ease13 of13 use13 Any13 relave13 or13 absolute13 me13 frame13 is13 saved13 as13 part13 of13 the13 search13 13
To13 save13 a13 search13
1 Set13 up13 and13 run13 the13 required13 syntax13 and13 me13 frame13
13
2 Click13 the13 Save13 Search13 icon13
13
3 Type13 in13 a13 name13 for13 the13 saved13 search13
Page of 24 71 Page of 24 71
13
7 To13 delete13 or13 update13 saved13 searches13 click13 13 manage13 saved13 searches13
13
Use13 the13 edit13 icons13 to13 make13 any13 changes13
Page of 26 71 Page of 26 71
13
Build13 simple13 Kibana13 histograms13 Kibana13 allows13 you13 to13 build13 charts13 and13 dashboards13 based13 on13 the13 search13 capability13 offered13 by13 the13 Discover13 page13 Saved13 searches13 can13 be13 used13 as13 the13 basis13 for13 charts13 and13 dashboards13 built13 using13 those13 charts13 To13 build13 a13 simple13 histogram13
1 Select13 the13 Visualize13 page13
13
2 Select13 Ver0cal13 Bar13 Chart13
13
3 To13 use13 a13 saved13 search13 click13 From13 a13 saved13 search13
Page of 27 71 Page of 27 71
13
4 Select13 a13 previously13 saved13 search13
13
5 Select13 to13 edit13 the13 x-shy‐axis
6 Edit13 the13 fields13 with13 informaon13 about13 the13 aggregaon13 field13 interval13 and13 custom13 label13
Page of 28 71 Page of 28 71
13
Page of 29 71 Page of 29 71
7 Click13 the13 green13 arrow13 The13 histogram13 is13 created13 and13 run13 based13 on13 the13 saved13 search
13
8 To13 save13 the13 histogram13 click13 the13 Save13 Visualiza0on13 icon13 and13 add13 a13 tle13
13
13
9 Click13 Save13
10 To13 view13 the13 visualisaon13 click13 the13 Dashboard13 page13
Page of 30 71 Page of 30 71
13
11 Click13 the13 Add13 Visualiza0on13 icon13
13
12 Select13 the13 visiualizaon13 that13 you13 just13 created13
13
Page of 31 71 Page of 31 71
13
13 To13 save13 the13 dashboard13 click13 the13 Save13 Dashboard13 icon13
Page of 32 71 Page of 32 71
13
14 Add13 a13 tle13 and13 select13 whether13 to13 store13 the13 selected13 me13 frame13 with13 the13 dashboard13
13
15 Click13 Save13
16 You13 can13 edit13 the13 shape13 and13 number13 of13 charts13 and13 save13 the13 dashboard13
17 To13 re-shy‐run13 the13 dashboard13 select13 the13 Dashboard13 page13 and13 click13 the13 Load13 Saved13 Dashboard13 icon13
Page of 33 71 Page of 33 71
13
18 Select13 the13 saved13 dashboard13 from13 the13 list13
13
19 The13 dashboard13 runs13
13
Page of 34 71 Page of 34 71
Page of 35 71 Page of 35 71
Aler0ng13 with13 ELK13 ELK13 examines13 the13 log13 records13 in13 process13 to13 send13 alerts13 to13 third-shy‐party13 alerng13 event13 management13 and13 social13 media13 applicaons13 As13 shown13 previously13 in13 the13 ELK13 architecture13 the13 ELK13 strategy13 for13 alerng13 is13 to13 send13 alerts13 from13 the13 Logstash13 component13 directly13 Therefore13 log13 records13 and13 alerts13 are13 sent13 to13 both13 Elascsearch13 and13 third-shy‐party13 tools13 at13 the13 same13 instant13 using13 the13 output13 secon13 of13 the13 Logstash13 configuraon13 Any13 alerts13 are13 generated13 during13 the13 filter13 secon13 processing13 of13 the13 Logstash13 configuraon13
Create13 alerts13 in13 Logstash13 The13 Logstash13 filter13 secon13 is13 used13 to13 trigger13 alerts13 based13 on13 the13 log13 records13 being13 processed13 For13 example13 log13 records13 can13 be13 checked13 for13 errors13 warnings13 crashed13 or13 stopped13 messages13 and13 generate13 an13 alert13
Here13 is13 an13 example13 Logstash13 filter13 code13
13 if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
The13 specific13 tags13 are13 set13 against13 the13 log13 records13 to13 trigger13 the13 alert13 later13 in13 the13 configuraon13 in13 the13 output13 secon13
Control13 the13 number13 of13 generated13 alerts13 using13 the13 Logstash13 throcle13 plugin13 so13 that13 either13 a13 lot13 of13 instances13 in13 a13 fixed13 me13 period13 are13 required13 before13 an13 alert13 is13 sent13 or13 a13 specific13 number13 of13 instances13 triggers13 the13 alert13
Page of 36 71 Page of 36 71
Edit13 the13 plugin13 parameters13 to13 meet13 system13 requirements13
13 13 13 13 13 if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 10
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle13
after_count =gt 20
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 30
period =gt 1200
key =gt servicename_alarm_warning
Page of 37 71 Page of 37 71
add_tag =gt servicename_throttled
13 13 13 13 13 13 13 13 13
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 40
period =gt 1500
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
Enrich13 alerts13 with13 message13 detail13 Some13 of13 the13 third-shy‐party13 integraons13 benefit13 from13 having13 components13 of13 the13 raw13 log13 record13 sent13 in13 addion13 to13 the13 raw13 record13 or13 fields13 in13 the13 record13 edited13 transformed13 or13 translated13
Grok13 Logstash13 allows13 the13 log13 record13 to13 be13 broken13 up13 into13 logical13 secons13 and13 each13 secon13 stored13 in13 a13 new13 parameter13 using13 the13 grok13 parameter13 in13 the13 filter13 secon13 Here13 is13 an13 example13 grok13
13 grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
To13 build13 the13 grok13 statement13 Logstash13 offers13 an13 online13 tool13 at13 hcpsgrokdebugherokuappcom13
Translate13 If13 specific13 fields13 in13 the13 log13 record13 are13 coded13 (for13 example13 the13 Bluemix13 app13 name13 is13 oaen13 not13 sent13 in13 the13 syslog13 messages13 instead13 the13 GUID13 is13 sent13 which13 is13 not13 useful)13 then13 the13 filter13 secon13 translate13 plugin13 can13 be13 used13 to13 add13 new13 fields13 by13 holding13 the13 decoded13 versions13 of13 those13 values13
13 translate
field =gt data1
destination =gt data1
override =gt true
Page of 38 71 Page of 38 71
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
dc50f6e4-c052-436a-bf27-95ba392c4228 monverifyservicemanagement
0c677e82-5bfc-4e38-9155-e416249e3ee6 ms-catalog-postpyretic-pomiculture
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
However13 the13 translate13 plugin13 is13 not13 included13 with13 the13 Logstash13 installaon13 In13 the13 next13 secon13 I13 will13 13 describe13 how13 to13 add13 new13 plugins13 to13 the13 Logstash13 instance13
Add13 custom13 plugins13 to13 Logstash13 As13 an13 open13 source13 tool13 Logstash13 encourages13 users13 to13 develop13 content13 for13 all13 users13 to13 share13 Some13 of13 the13 plugins13 that13 were13 created13 by13 our13 community13 include13 a13 translaon13 plugin13 an13 output13 Slack13 plugin13 and13 an13 output13 syslog13 plugin13 Letrsquos13 look13 at13 how13 to13 add13 these13 plugins13 to13 your13 environment13
Filter13 translate13 plugin13 To13 add13 the13 translate13 plugin13
1 Download13 the13 plugin13
2 Copy13 the13 plugin13 file13 logstash-shy‐filter-shy‐translate-shy‐214gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-filter-translate 214
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Output13 Slack13 plugin13 To13 add13 the13 Slack13 plugin13
Page of 39 71 Page of 39 71
1 Download13 the13 plugin13 13
2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐slack-shy‐014gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-output-slack 014
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Output13 syslog13 plugin13 To13 add13 the13 syslog13 plugin13
1 Download13 the13 plugin13 13
2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐syslog-shy‐214gem13 to13 the13 ELK13 Virtual13 Server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-output-syslog 214
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Integrate13 Logstash13 with13 Slack13 To13 send13 generated13 alerts13 to13 Slack13 complete13 these13 steps13
1 Create13 a13 Slack13 integraon13 13
2 Use13 the13 Slack13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 Slack13 integraon13
113 Login13 to13 Slack13 and13 go13 to13 the13 Apps13 page13 13
Page of 40 71 Page of 40 71
13
313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13
13
A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13
13
Page of 42 71 Page of 42 71
The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13
13
The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13
13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
slack
url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5
channel =gt cloud-operations-mon
username =gt nick_cawood
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13
Page of 43 71 Page of 43 71
Page of 44 71 Page of 44 71
In13 Slack13 the13 alerts13 look13 like13 this13
13
Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13
1 Create13 a13 PagerDuty13 integraon13 13
2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13
1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13
13
2 Click13 Add13 New13 Service13
13
313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13
Page of 45 71 Page of 45 71
13
413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13
513 Click13 Add13 Service13
13
The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13
Page of 46 71 Page of 46 71
13
The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt be7eb1ea9c224018923e818d3e891d0b
incident_key =gt logstashhosttype
Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13
Page of 47 71 Page of 47 71
13
Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13
1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13
2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13
3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13
4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13
13 13 http
url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo
http_method =gt post
format =gt json
For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13
13 13 syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
Page of 48 71 Page of 48 71
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13
13
The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13
if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2
This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13
Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13
Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13
Logstash13 severity Shown13 in13 Omnibus13 as
alert Indeterminate
crical Warning
debug 7
emergency Clear
error Minor
informaonal 6
noce Crical
warning Major
Page of 49 71 Page of 49 71
13
Page of 50 71 Page of 50 71
Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13
input
tcp
port =gt 5000
type =gt syslog
filter
mutate
add_field =gt format =gt cf
grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
translate
field =gt data1
destination =gt data1
override =gt true
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
=== Check Alarm Criteria ===
Standard Key Words
if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
=== Alarm Exclusions ===
if [message] =~ (i)sometexthere
mutate remove_tag =gt servicename_alarm
Throttle alarms so we do not hit PD rate limits
Page of 52 71 Page of 52 71
if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 5
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle
after_count =gt 5
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 5
period =gt 1200
key =gt servicename_alarm_warning
add_tag =gt servicename_throttled
Page of 53 71 Page of 53 71
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
output
elasticsearch
hosts =gt localhost
stdout codec =gt rubydebug
if servicename_alarm_err in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
Page of 54 71 Page of 54 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 55 71 Page of 55 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
Page of 56 71 Page of 56 71
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert err message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_error in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
Page of 57 71 Page of 57 71
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 58 71 Page of 58 71
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 59 71 Page of 59 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert error message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
Page of 60 71 Page of 60 71
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
Page of 61 71 Page of 61 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 62 71 Page of 62 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt x
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert fatal message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
Page of 63 71 Page of 63 71
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
Page of 64 71 Page of 64 71
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 65 71 Page of 65 71
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt heavy_exclamation_mark
Page of 66 71 Page of 66 71
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert WARNING message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 67 71 Page of 67 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
Page of 68 71 Page of 68 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 69 71 Page of 69 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
Page of 70 71 Page of 70 71
Page of 71 71 Page of 71 71
Introduc0on13 IBMreg13 Bluemixreg13 IBMrsquos13 PlaRorm13 as13 a13 Service13 (PaaS)13 allows13 you13 to13 rapidly13 build13 and13 deploy13 an13 applicaon13 Applicaons13 can13 run13 in13 a13 nave13 or13 a13 hybrid13 Bluemix13 environment13 13 13
This13 document13 describes13 the13 integraon13 of13 the13 Elascsearch13 Logstash13 Kibana13 (ELK)13 log13 analycs13 open13 source13 stack13 into13 the13 Bluemix13 environment13 how13 to13 collect13 logs13 from13 Bluemix13 applicaons13 (apps)13 and13 send13 alerts13 from13 those13 log13 records13 to13 selected13 alertevent13 noficaon13 targets13 (PagerDuty13 Slack13 and13 IBM13 Netcool13 Omnibus)13
This13 document13 describes13 how13 to13 provision13 Bluemix13 virtual13 servers13 in13 the13 Bluemix13 environment13 install13 the13 ELK13 stack13 on13 a13 Bluemix13 virtual13 server13 and13 configure13 the13 ELK13 stack13 to13 integrate13 with13 other13 components13 deployed13 on13 the13 Bluemix13 environment13
The13 steps13 to13 integrate13 the13 ELK13 stack13 into13 the13 Cloud13 Architecture13 and13 Soluon13 Engineering13 (CASE)13 environment13 are13 used13 as13 the13 basis13 for13 this13 document13 13 For13 reference13 both13 the13 ELK13 architecture13 within13 the13 CASE13 environment13 and13 the13 whole13 CASE13 environment13 are13 defined13 using13 system13 architecture13 diagrams13 as13 well13 as13 diagrams13 defining13 the13 ELK13 structure13 and13 ELK13 integraon13 into13 Bluemix13 architecture13
What13 is13 Elas0csearch13 Logstash13 Kibana13 Elascsearch13 Logstash13 Kibana13 is13 an13 open13 source13 stack13 of13 three13 applicaons13 that13 work13 together13 to13 provide13 an13 end-shy‐to-shy‐end13 search13 and13 visualisaon13 plaRorm13 for13 the13 invesgaon13 of13 log13 file13 sources13 in13 real13 me13 Log13 file13 records13 can13 be13 searched13 from13 a13 variety13 of13 log13 sources13 charts13 and13 dashboards13 built13 to13 visualize13 the13 log13 records13 The13 three13 applicaons13 are13 Elascsearch13 Logstash13 and13 Kibana13
Logstash13 Logstash13 is13 an13 applicaon13 that13 allows13 the13 ELK13 stack13 to13 manage13 the13 gathering13 transformaon13 and13 transport13 of13 log13 file13 records13 Logstash13 is13 able13 to13 pull13 records13 from13 remote13 sources13 or13 listen13 for13 records13 on13 specified13 ports13
Logstash13 has13 three13 areas13 of13 funcon13
bull Process13 input13 of13 the13 log13 file13 records13 from13 the13 remote13 source13 You13 can13 configure13 Logstash13 to13 receive13 records13 in13 a13 variety13 of13 methods13 for13 example13 reading13 from13 a13 flat13 file13 or13 from13 a13 TCP13 port13 or13 direct13 from13 a13 remote13 or13 local13 syslog13 These13 methods13 are13 defined13 in13 the13 Logstash13 online13 support13 and13 input13 secon13 of13 the13 Logstash13 conf13 file13 Once13 all13 input13 processing13 is13 completed13 Logstash13 moves13 onto13 the13 next13 funcon13
bull Transform13 and13 enrich13 the13 log13 records13 that13 take13 place13 in13 the13 Logstash13 filter13 secon13 Any13 number13 of13 changes13 to13 the13 format13 (and13 content13 if13 required)13 of13 the13 log13 file13 record13 can13 be13 made13 using13 the13 various13 methods13 provided13 with13 Logstash13 for13 example13 methods13 to13 add13 and13 enrich13 log13 record13 fields13 and13 the13 flagging13 I13 describe13 excluding13 and13 liming13 records13 for13 alerng13 later13 in13 this13 document13 Aaer13 the13 log13 record13 is13 transformed13 Logstash13 processes13 the13 record13
bull Send13 the13 log13 records13 to13 the13 Elascsearch13 applicaon13 from13 the13 output13 secon13 of13 the13 Logstash13 configuraon13 A13 specific13 elascsearch13 method13 is13 included13 with13 the13 Logstash13 instance13 that13 sends13 the13 log13 records13 to13 the13 Elascsearch13 instance13 However13 as13 with13 the13 input13 and13 filter13
Page of 3 71 Page of 3 71
secons13 a13 variety13 of13 methods13 are13 available13 to13 process13 the13 data13 sent13 from13 the13 filter13 secon13 For13 example13 the13 records13 can13 be13 sent13 to13 external13 alerng13 tools13 such13 as13 PagerDuty13 social13 media13 tools13 such13 as13 Slack13 or13 event13 management13 hubs13 like13 IBM13 Netcool13 Omnibus13 via13 output13 methods13 syslog13 or13 hcp13 13
As13 an13 open13 source13 tool13 non-shy‐Logstash13 employees13 develop13 other13 methods13 and13 share13 them13 with13 the13 user13 community13 for13 addion13 to13 the13 library13 of13 methods13 Logstash13 can13 execute13 (examples13 of13 this13 process13 are13 included13 later13 in13 this13 document)13
Elas0csearch13 Elascsearch13 is13 the13 index13 store13 and13 query13 applicaon13 on13 the13 ELK13 stack13 It13 provides13 a13 standard13 full13 text13 search13 facility13 of13 the13 log13 file13 records13 processed13 by13 Logstash13 The13 ELK13 online13 help13 documentaon13 describes13 Elascsearch13 as13 follows13 ldquoIt13 is13 generally13 used13 as13 the13 underlying13 enginetechnology13 that13 powers13 applicaons13 that13 have13 complex13 search13 features13 and13 requirementsrdquo13
Kibana13 Kibana13 is13 the13 user13 interface13 for13 the13 ELK13 stack13 Kibana13 integrates13 with13 Elascsearch13 to13 visualise13 the13 results13 of13 searches13 and13 allow13 the13 creaon13 of13 charts13 and13 dashboards13 based13 on13 the13 searches13 The13 searches13 charts13 and13 dashboard13 can13 visualise13 real-shy‐me13 or13 historical13 log13 records13 Users13 access13 Kibana13 through13 an13 hcp13 URL13 from13 a13 client13 web13 browser13
ELK13 System13 Architecture13 The13 ELK13 stack13 can13 be13 described13 as13 follows13
Page of 4 71 Page of 4 71
13
Page of 5 71 Page of 5 71
Service13 Management13 architecture13 The13 IBM13 Cloud13 Architecture13 Center13 created13 a13 standard13 reference13 architecture13 for13 how13 to13 set13 up13 your13 cloud13 environemnt13 to13 handle13 incident13 management13 13
View13 the13 standard13 Service13 Management13 architecure13
13 13
Log13 Monitoring13 System13 architecture13 The13 following13 diagram13 shows13 a13 standard13 architecture13 for13 the13 log-shy‐monitoring13 system13 described13 in13 this13 tutorial13 13
13
Bluemix13 environment13 system13 architecture13 The13 ELK13 stack13 can13 be13 installed13 into13 a13 variety13 of13 environments13 When13 using13 Bluemix13 the13 following13 figure13 shows13 what13 the13 environment13 might13 look13 like13
Page of 6 71 Page of 6 71
13
The13 Bluemix13 cloud13 plaRorm13 offers13 a13 Public13 network13 and13 access13 to13 US13 South13 and13 UK13 environments13 From13 all13 these13 environments13 third-shy‐party13 cloud-shy‐based13 soluons13 are13 also13 accessible13
The13 ELK13 stack13 is13 installed13 in13 the13 Bluemix13 United13 Kingdom13 environment13 because13 this13 is13 the13 environment13 where13 Bluemix13 has13 virtual13 servers13 available13 for13 provisioning13 Other13 applicaons13 in13 the13 CASE13 project13 were13 also13 installed13 on13 virtual13 servers13 in13 the13 Bluemix13 United13 Kingdom13 region13
Bluemix13 apps13 were13 used13 to13 create13 a13 system13 to13 be13 monitored13 by13 ELK13 These13 apps13 were13 hosted13 in13 the13 Bluemix13 US13 South13 environment13 using13 Bluemix13 microservices13 which13 produce13 logs13 during13 normal13 operaon13 and13 during13 crashes13 and13 outages13
Through13 the13 Bluemix13 Cloud13 Foundry13 process13 the13 log13 records13 can13 be13 broadcast13 from13 the13 Bluemix13 apps13 so13 Logstash13 can13 receive13 them13 and13 send13 them13 to13 Elascsearch13 within13 the13 ELK13 stack13 or13 other13 virtual13 servers13 (hosng13 other13 IBM13 or13 third-shy‐party13 products)13 in13 the13 Bluemix13 United13 Kingdom13 environment13 or13 to13 third-shy‐party13 applicaons13 in13 the13 public13 cloud13 (Pager13 Duty13 Slack13 etc)13
Crea0ng13 a13 virtual13 server13 in13 Bluemix13 With13 Bluemix13 you13 can13 provision13 a13 virtual13 server13 in13 the13 cloud13 to13 install13 standalone13 or13 integrated13 system13 components13 Not13 all13 Bluemix13 regions13 offer13 this13 service13 all13 the13 me13 due13 to13 beta-shy‐project13 constraints13 In13 this13 arclersquos13 examples13 virtual13 servers13 were13 allowed13 to13 be13 provisioned13 in13 the13 United13 Kingdom13 region13 For13 more13 informaon13 read13 how13 to13 provision13 a13 Bluemix13 virtual13 server13 including13 some13 simple13 setup13 steps13 for13 ease13 of13 use13
Page of 7 71 Page of 7 71
Create13 an13 SSH13 key13 To13 access13 your13 virtual13 server13 once13 it13 is13 provisioned13 allocate13 an13 SSH13 key13 to13 the13 virtual13 server13
To13 generate13 a13 publicprivate13 RSA13 key13 pair13 follow13 these13 steps13
1 In13 a13 UNIX13 or13 Linuxreg13 window13 run13 the13 following13 command13 specifying13 a13 ltuniquekeynamegt13 of13 your13 your13 choice13 and13 a13 password13 or13 passphrase13 (memorize13 the13 password)$13 ssh-shy‐keygen13 -shy‐t13 rsa13 -shy‐f13 ltuniquekeynamegt13
2 Enter13 the13 passphrase13 (empty13 for13 no13 passphrase)13
3 Enter13 the13 same13 passphrase13 again13
Your13 idenficaon13 has13 been13 saved13 in13 ltuniquekeynamegt13
Your13 public13 key13 has13 been13 saved13 in13 ltuniquekeynamegtpub13
The13 key13 fingerprint13 is13
d6268464506525930ce4632c693c203c13 ltusergtltservergt13
The13 keys13 randomart13 image13 is13
+--[ RSA 2048]----+
|o ++=o |
| E o ooo |
| |
| + o |
| S o |
| o |
| |
| |
| |
+-----------------+
Two13 keys13 are13 created13 public13 and13 private13 The13 public13 key13 contents13 are13 used13 to13 create13 a13 virtual13 server13 with13 a13 public13 IP13 address13 The13 private13 key13 is13 used13 to13 access13 the13 virtual13 server13 from13 a13 client13 machine13
Page of 8 71 Page of 8 71
Provisioning13 a13 virtual13 server13 1 A13 Bluemix13 account13 is13 required13 for13 this13 process13 Sign13 up13 here13 and13 log13 in13 Make13 sure13 your13
environment13 is13 correct13 For13 example13 if13 you13 are13 in13 the13 United13 Kingdom13 select13 the13 United13 Kingdom13 region13 in13 the13 Bluemix13 console13
13
13
2 Add13 a13 virtual13 server13 to13 your13 Bluemix13 environment13 Click13 Run13 Virtual13 Servers13
13
3 Select13 an13 OS13 image13 from13 the13 EXISTING13 list13 For13 ELK13 select13 Centos13 OS13 (use13 latest13 available13 version)13
Page of 9 71 Page of 9 71
4 Name13 the13 virtual13 server13
5 Make13 sure13 Assign13 public13 IP13 address13 is13 checked13
13
6 Click13 Add13 Key13 to13 add13 the13 SSH13 key13 you13 created13 previously13 Copy13 the13 contents13 of13 the13 public13 key13 into13 the13 Public13 Key13 to13 import13 field13
Page of 10 71 Page of 10 71
13
7 Click13 OK13 Then13 click13 CREATE13
The13 virtual13 server13 is13 provisioned13 and13 added13 to13 the13 Bluemix13 dashboard13 under13 VIRTUAL13 SERVERS13
Page of 11 71 Page of 11 71
13
Set13 up13 the13 virtual13 server13
Log13 on13 to13 the13 virtual13 server13 Now13 you13 can13 log13 on13 to13 the13 virtual13 server13 and13 add13 new13 users13 and13 programs13 to13 make13 it13 easier13 to13 access13 the13 virtual13 server13
To13 access13 the13 server13 go13 to13 the13 UNIX13 or13 Linux13 window13 that13 you13 used13 to13 create13 the13 SSH13 key13 In13 the13 directory13 where13 the13 ltuniquekeynamegt13 files13 exist13 run13 the13 following13 command13 referencing13 the13 SSH13 keyname13 the13 public13 IP13 address13 Bluemix13 allocated13 for13 the13 virtual13 server13 and13 the13 password13 for13 the13 SSH13 key13
[userserver ~]$ ssh -i ltuniquekeynamegt ibmcloudltpublicIPaddressgt
Enter13 the13 passphrase13 for13 the13 key13 ltuniquekeynamegt13
Last login Tue Jul 5 125546 2016 from 108-210-96-231lightspeedrlghncsbcglobalnet
[ibmcloudelkforcase ~]$
ibmcloud13 is13 the13 default13 user13 for13 the13 Bluemix13 virtual13 servers13 and13 allows13 SSH13 to13 access13 the13 server13
Acces13 the13 root13 user13 To13 access13 the13 root13 user13 log13 on13 as13 ibmcloud13 user13 and13 run13 the13 following13 command13
[ibmcloudelkforcase ~]$ sudo bash
[rootelkforcase ibmcloud]
The13 root13 user13 can13 be13 used13 to13 add13 programs13 to13 the13 server13 and13 create13 and13 edit13 users13 as13 required13 to13 access13 applicaons13 installed13 on13 the13 virtual13 server13
Page of 12 71 Page of 12 71
Copy13 files13 to13 the13 virtual13 server13 To13 move13 installaon13 files13 to13 the13 server13 go13 to13 the13 UNIX13 or13 Linux13 window13 you13 used13 to13 create13 the13 SSH13 key13 and13 the13 directory13 where13 the13 ltuniquekeynamegt13 files13 exist13 Move13 the13 installaon13 files13 to13 the13 same13 locaon13
Run13 the13 following13 command13 referencing13 the13 SSH13 keyname13 the13 public13 IP13 address13 Bluemix13 allocated13 for13 the13 virtual13 server13 and13 the13 password13 for13 the13 SSH13 key13
[userserver ~]$ scp ndashi ltuniquekeynamegt ltINSTALLATIONFILENAMEgt ibmcloudltpublicIPaddressgthomeibmcloud
Enter13 the13 passphrase13 for13 key13 cloudkey313
This13 copies13 the13 file13 using13 the13 ibmcloud13 user13 to13 the13 homeibmcloud13 directory13 on13 the13 virtual13 server13
Install13 and13 access13 the13 virtual13 server13 via13 VNC13 server13 Using13 the13 ssh13 command13 to13 access13 the13 virtual13 server13 is13 not13 the13 most13 stable13 method13 and13 does13 not13 offer13 X-shy‐Windows13 support13 Therefore13 we13 suggest13 that13 you13 use13 a13 VNC13 client13 to13 access13 the13 virtual13 server13
Installing13 a13 VNC13 client13 should13 be13 done13 as13 the13 root13 user13 Use13 the13 yum13 command13 as13 follows13
[ibmcloudelkforcase ~]$ yum groupinstall GNOME Desktop
[ibmcloudelkforcase ~]$ yum install tigervnc-server
These13 commands13 first13 install13 a13 desktop13 style13 OS13 user13 interface13 and13 then13 a13 VNC13 client13 to13 remotely13 access13 the13 desktop13 style13 OS13
Then13 as13 either13 the13 root13 user13 or13 another13 user13 (example13 below13 uses13 elk)13 run13 the13 following13 command13 to13 create13 the13 VNC13 client13 connecon13
[elkelkforcase ~]$ vncserver
New elkforcase2 (elk) desktop is elkforcase2
Starting applications specified in homeelkvncxstartup
Log file is homeelkvncelkforcase2log
[elkelkforcase ~]$
Install13 a13 VNC13 client13 on13 the13 client13 machine13 used13 to13 access13 the13 Internet13 and13 (in13 the13 example13 above)13 use13 the13 command13 ltpublicIPadddressgt213 and13 the13 password13 for13 ELK13 to13 access13 the13 virtual13 server13
Using13 a13 VNC13 client13 means13 that13 you13 can13 leave13 long13 duraon13 installaons13 running13 without13 fear13 of13 the13 terminal13 losing13 connecon13 and13 the13 installaons13 failing13
Install13 the13 ELK13 stack13 Download13 the13 source13 soaware13 code13 for13 the13 ELK13 stack13 from13 the13 ELK13 website13 13
At13 the13 me13 of13 wring13 the13 downloaded13 soaware13 files13 were13
Page of 13 71 Page of 13 71
13
Transfer13 the13 files13 to13 the13 virtual13 server13 (using13 the13 scp13 method13 detailed13 above)13
Install13 Elas0csearch13 As13 a13 root13 user13 create13 a13 non-shy‐root13 user13 with13 the13 adduser13 command13
Create13 a13 directory13 to13 install13 Elascsearch13 into13 and13 untar13 the13 contents13 of13 the13 Elascsearch13 install13 file13 into13 that13 directory13
[elkelkforcase]$ cd home
[elkelkforcase]$ mkdir elasticsearch
[elkelkforcase]$ cp homeibmcloudelasticsearch-233targz homeelasticsearch
[elkelkforcase]$ cd elasticsearch
[elkelkforcase]$ gunzip elasticsearch-233targz
[elkelkforcase]$ untar elasticsearch-233tar
Elascsearch13 is13 now13 installed13
Start13 Elas0csearch13 To13 start13 Elascsearch13 go13 to13 the13 bin13 directory13 as13 the13 non-shy‐root13 user13 when13 you13 installed13 it13 and13 run13 the13 elascsearch13 executable13
[elkelkforcase]$ cd homeelasticsearchelasticsearch-233bin
[elkelkforcase]$ elasticsearch amp
[2] 22211
[elkelkforcase]$ [2016-07-05 133857730][INFO ][node] [Arsenal] version[233] pid[22211] build[218bdf12016-05-17T154004Z]
[2016-07-05 133857740][INFO ][node] [Arsenal] initializing
[2016-07-05 133858984][INFO ][plugins] [Arsenal] modules [reindex lang-expression lang-groovy] plugins [] sites []
[2016-07-05 133859146][INFO ][env] [Arsenal] using [1] data paths mounts [[ (rootfs)]] net usable_space [155gb] net total_space [199gb] spins [unknown] types [rootfs]
[2016-07-05 133859146][INFO ][env] [Arsenal] heap size [10156mb] compressed ordinary object pointers [true]
Page of 14 71 Page of 14 71
[2016-07-05 133859147][WARN ][env] [Arsenal] max file descriptors [4096] for elasticsearch process likely too low consider increasing to at least [65536]
[2016-07-05 133903526][INFO ][node] [Arsenal] initialized
[2016-07-05 133903526][INFO ][node] [Arsenal] starting
To13 test13 that13 Elascsearch13 is13 running13 correctly13 open13 the13 browser13 in13 the13 VNC13 client13 and13 run13 localhost920013
13
Install13 Kibana13 ELK13 recommends13 installing13 and13 running13 Kibana13 as13 root13 13
1 As13 root13 user13 create13 an13 installaon13 directory13 and13 untar13 the13 installaon13 file13
[rootelkforcase]$ cd home
[rootelkforcase]$ mkdir kibana
[rootelkforcase]$ cp homeibmcloudkibana-451-linux-x64targz homekibana
[rootelkforcase]$ cd kibana
[rootelkforcase]$ gunzip kibana-451-linux-x64targz
[rootelkforcase]$ untar kibana-451-linux-x64tar
2 Modify13 the13 manifestyml13 configuraon13 file13 Go13 to13 the13 kibana-shy‐451-shy‐linux-shy‐x6413 directory13
[rootelkforcase]$ cd homekibanakibana-451-linux-x64
3 Edit13 the13 manifestyml13 file13 so13 the13 contents13 are13
applicaons13
-shy‐ name13 kibana-shy‐in-shy‐bluemix13
memory13 128M13
host13 kibana-shy‐in-shy‐bluemix13
buildpack13 hcpsgithubcomcloudfoundry-shy‐communitynginx-shy‐buildpackgit13
Page of 15 71 Page of 15 71
4 Edit13 the13 configkibanayml13 file13 so13 the13 contents13 are13
Kibana is served by a back end server This controls which port to use
port 5601
The host to bind the server to
host 0000
If you are running kibana behind a proxy and want to mount it at a path
specify that path here The basePath cant end in a slash
serverbasePath
The maximum payload size in bytes on incoming server requests
servermaxPayloadBytes 1048576
The Elasticsearch instance to use for all your queries
elasticsearchurl httplocalhost9200
preserve_elasticsearch_host true will send the hostname specified in `elasticsearch` If you set it to false
then the host you use to connect to this Kibana instance will be sent
elasticsearchpreserveHost true
Kibana uses an index in Elasticsearch to store saved searches visualizations
and dashboards It will create a new index if it doesnt already exist
kibanaindex kibana
The default application to load
kibanadefaultAppId discover
If your Elasticsearch is protected with basic auth these are the user credentials
used by the Kibana server to perform maintenance on the kibana_index at startup Your
Kibana users will still need to authenticate with Elasticsearch (which is proxied through
the Kibana server)
Page of 16 71 Page of 16 71
elasticsearchusername user
elasticsearchpassword pass
SSL for outgoing requests from the Kibana Server to the browser (PEM formatted)
serversslcert pathtoyourservercrt
serversslkey pathtoyourserverkey
Optional setting to validate that your Elasticsearch backend uses the same key files (PEM formatted)
elasticsearchsslcert pathtoyourclientcrt
elasticsearchsslkey pathtoyourclientkey
If you need to provide a CA certificate for your Elasticsearch instance put
the path of the pem file here
elasticsearchsslca pathtoyourCApem
Set to false to have a complete disregard for the validity of the SSL
certificate
elasticsearchsslverify true
Time in milliseconds to wait for elasticsearch to respond to pings defaults to
request_timeout setting
elasticsearchpingTimeout 1500
Time in milliseconds to wait for responses from the back end or elasticsearch
This must be gt 0
elasticsearchrequestTimeout 30000
Time in milliseconds for Elasticsearch to wait for responses from shards
Set to 0 to disable
elasticsearchshardTimeout 0
Time in milliseconds to wait for Elasticsearch at Kibana startup before retrying
elasticsearchstartupTimeout 5000
elasticsearchstartupTimeout 30000
Page of 17 71 Page of 17 71
Set the path to where you would like the process id file to be created
pidfile varrunkibanapid
If you would like to send the log output to a file you can set the path below
loggingdest stdout
Set this to true to suppress all logging output
loggingsilent false
Set this to true to suppress all logging output except for error messages
loggingquiet false
Set this to true to log all events including system usage information and all requests
loggingverbose false
Kibana13 is13 now13 installed13
Start13 Kibana13 To13 start13 Kibana13 go13 to13 the13 bin13 directory13 and13 run13 the13 kibana13 executable13
[rootelkforcase]$
[rootelkforcase]$ cd homekibanakibana-451-linux-x64bin
[rootelkforcase]$ kibana amp
[1] 22283
[rootelkforcase] log [095900542] [info][status][pluginkibana] Status changed from uninitialized to green - Ready
log [095900600] [info][status][pluginelasticsearch] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [095900689] [info][status][pluginkbn_vislib_vis_types] Status changed from uninitialized to green - Ready
log [095900717] [info][status][pluginmarkdown_vis] Status changed from uninitialized to green - Ready
log [095900721] [info][status][pluginmetric_vis] Status changed from uninitialized to green - Ready
log [095900814] [info][status][pluginspyModes] Status changed from uninitialized to green - Ready
log [095900934] [info][status][pluginstatusPage] Status changed from uninitialized to green - Ready
Page of 18 71 Page of 18 71
log [095900941] [info][status][plugintable_vis] Status changed from uninitialized to green - Ready
log [095901067] [info][listening] Server running at http00005601
log [095906083] [info][status][pluginelasticsearch] Status changed from yellow to yellow - No existing Kibana index found
log [095908942] [info][status][pluginelasticsearch] Status changed from yellow to green - Kibana index ready
Kibana13 is13 now13 started13
Broadcast13 Bluemix13 app13 log13 records13 Bluemix13 apps13 (which13 are13 shown13 in13 the13 US13 South13 Bluemix13 region13 of13 the13 System13 Architecture13 diagram)13 will13 produce13 logs13 in13 a13 syslog13 rfc542413 format13 using13 a13 Bluemix13 tool13 called13 Loggregator13
For13 Logstash13 to13 receive13 these13 logs13 each13 Bluemix13 app13 needs13 to13 broadcast13 these13 logs13 To13 do13 that13 run13 the13 following13 commands13 on13 each13 Bluemix13 app13 using13 the13 public13 IP13 address13 of13 the13 ELK13 virtual13 server13 the13 ELK13 tag13 to13 trace13 the13 process13 and13 insert13 the13 name13 of13 each13 app13 in13 turn13
cf cups elk -l syslogltpublicIPaddress5000 cf bind-service ltmyappgt elk
cf restage ltmyappgt
Install13 Logstash13 To13 install13 Logstash13 as13 root13 user13 create13 a13 directory13 for13 the13 installaon13 and13 untar13 the13 installaon13 file13 there13
13 13 [rootelkforcase]$ cd home
[rootelkforcase]$ mkdir logstash
[rootelkforcase]$ cp homeibmcloudlogstash-233targz homelogstash
[rootelkforcase]$ cd logstash
[rootelkforcase]$ gunzip logstash-233targz
[rootelkforcase]$ untar logstash-233tar
Write13 Logstash13 configura0on13 Logstash13 requires13 a13 configuraon13 file13 to13 be13 wricen13 to13 receive13 transform13 and13 output13 the13 log13 records13 to13 Elascsearch13 This13 secon13 explains13 the13 basic13 requirements13 of13 processing13 Bluemix13 app13 log13 records13 into13 Elascsearch13 (more13 complex13 configuraons13 are13 detailed13 later13 in13 this13 document13 for13 the13 integraons13 with13 third-shy‐party13 tools)13
Create13 a13 conf13 file13 as13 root13 user13
[rootelkforcase]$ cd homelogstashlogstash-233
[rootelkforcase]$ mkdir config
Page of 19 71 Page of 19 71
[rootelkforcase]$ cd config
[rootelkforcase]$ touch loggregatorconf
Edit13 the13 loggregator13 file13 with13 the13 following13 contents13
13 Input13 secon13 defines13 which13 log13 records13 to13 receive13
13 port13 and13 type13 reflect13 method13 selected13 for13 Cloud13 Foundry13 to13 broadcast13 the13 records13
input
tcp
port =gt 5000
type =gt syslog
13 Filter13 secon13 ndash13 transforms13 the13 data13
filter
Add a field to identify the records as Cloud Foundry based
mutate
add_field =gt format =gt cf
13 Output13 secon13 ndash13 sends13 the13 data13 to13 its13 desnaon(s)13
output
13 Use13 the13 elascsearch13 plugin13 to13 send13 the13 data13 to13 the13 elascsearch13 instance13
13 elasticsearch
hosts =gt localhost
stdout codec =gt rubydebug
13 End13 of13 Config13
Save13 the13 file13 Verify13 the13 format13 of13 the13 file13 is13 acceptable13 by13 running13 the13 following13 commands13
[rootelkforcase]$ cd homelogstashlogstash-233
[rootelkforcase]$ binlogstash agent --configtest -f configloggregatorconf
Page of 20 71 Page of 20 71
Configuraon13 is13 OK13
Start13 Logstash13 Aaer13 you13 have13 wricen13 and13 verified13 the13 Logstash13 configuraon13 you13 can13 start13 Logstash13 to13 process13 records13 from13 the13 Bluemix13 apps13 (via13 Cloud13 Foundry)13 Run13 the13 following13 as13 root13 user13
[rootelkforcase]$ cd homelogstashlogstash-233
[rootelkforcase]$ binlogstash agent -f configloggregatorconf
Logstash13 is13 now13 started13
Monitor13 Logstash13 processing13 Once13 Logstash13 is13 running13 and13 log13 records13 are13 being13 produced13 by13 the13 Bluemix13 apps13 you13 can13 monitor13 progress13 in13 the13 same13 window13 that13 was13 used13 to13 start13 Logstash13 Output13 is13 similar13 to13 the13 following13
13
The13 log13 records13 being13 broadcast13 from13 the13 Bluemix13 apps13 are13 now13 sent13 to13 Elascsearch13 from13 the13 Logstash13 output13 configuraon13 Records13 are13 sent13 in13 real13 me13 If13 Logstash13 is13 not13 running13 while13 log13 records13 are13 being13 produced13 then13 these13 records13 will13 not13 be13 processed13 by13 Logstash13 and13 will13 not13 be13 stored13 in13 Elascsearch13
Page of 21 71 Page of 21 71
Use13 ELK13 The13 log13 records13 from13 the13 Bluemix13 apps13 are13 now13 being13 stored13 and13 indexed13 by13 the13 Elascsearch13 component13 This13 data13 can13 be13 accessed13 via13 the13 Kibana13 applicaon13
Kibana13 Discover13 page13 Using13 a13 web13 browser13 either13 directly13 from13 a13 client13 or13 using13 the13 VNC13 client13 access13 Kibana13 using13 the13 publicIPaddress13 of13 the13 ELK13 virtual13 server13
hcpltpublicIPaddressgt560113
No13 userpassword13 credenals13 are13 required13
Kibana13 defaults13 to13 a13 search13 of13 all13 records13 ()13 in13 the13 last13 1513 minutes13 and13 returns13 the13 results13
13
More13 complex13 searches13 are13 possible13 by13 changing13 the13 me13 filter13 and13 supplying13 search13 queries13
Change13 the13 0me13 filter13 To13 change13 the13 me13 ranges13 click13 the13 clock13 symbol13 in13 the13 top-shy‐right13 of13 the13 Kibana13 Discover13 page13
13
You13 will13 see13 the13 following13 opons13
13
Page of 22 71 Page of 22 71
A13 variety13 of13 quick13 me13 ranges13 can13 be13 selected13 or13 you13 can13 switch13 to13 relave13 or13 absolute13 me13 ranges13
Quick13 me13 ranges13 allow13 switching13 between13 last13 X13 minutes13 to13 last13 Y13 hours13 or13 Z13 days13 etc13
Relave13 me13 ranges13 allow13 more13 specific13 versions13 of13 the13 Quick13 opons13 from13 specific13 minuteshoursdays13 to13 now13
13
Absolute13 me13 ranges13 specify13 the13 start13 and13 end13 mes13 for13 the13 searches13
13
Custom13 search13 queries13 You13 can13 use13 the13 text13 window13 of13 the13 Discover13 page13 to13 search13 for13 matches13 within13 the13 log13 records13 You13 can13 use13 Apache13 Lucene13 or13 Elascsearch13 query13 DSL13 syntax13
13
Page of 23 71 Page of 23 71
Saved13 searches13 You13 can13 save13 any13 custom13 search13 syntax13 and13 choice13 of13 me13 range13 for13 future13 ease13 of13 use13 Any13 relave13 or13 absolute13 me13 frame13 is13 saved13 as13 part13 of13 the13 search13 13
To13 save13 a13 search13
1 Set13 up13 and13 run13 the13 required13 syntax13 and13 me13 frame13
13
2 Click13 the13 Save13 Search13 icon13
13
3 Type13 in13 a13 name13 for13 the13 saved13 search13
Page of 24 71 Page of 24 71
13
7 To13 delete13 or13 update13 saved13 searches13 click13 13 manage13 saved13 searches13
13
Use13 the13 edit13 icons13 to13 make13 any13 changes13
Page of 26 71 Page of 26 71
13
Build13 simple13 Kibana13 histograms13 Kibana13 allows13 you13 to13 build13 charts13 and13 dashboards13 based13 on13 the13 search13 capability13 offered13 by13 the13 Discover13 page13 Saved13 searches13 can13 be13 used13 as13 the13 basis13 for13 charts13 and13 dashboards13 built13 using13 those13 charts13 To13 build13 a13 simple13 histogram13
1 Select13 the13 Visualize13 page13
13
2 Select13 Ver0cal13 Bar13 Chart13
13
3 To13 use13 a13 saved13 search13 click13 From13 a13 saved13 search13
Page of 27 71 Page of 27 71
13
4 Select13 a13 previously13 saved13 search13
13
5 Select13 to13 edit13 the13 x-shy‐axis
6 Edit13 the13 fields13 with13 informaon13 about13 the13 aggregaon13 field13 interval13 and13 custom13 label13
Page of 28 71 Page of 28 71
13
Page of 29 71 Page of 29 71
7 Click13 the13 green13 arrow13 The13 histogram13 is13 created13 and13 run13 based13 on13 the13 saved13 search
13
8 To13 save13 the13 histogram13 click13 the13 Save13 Visualiza0on13 icon13 and13 add13 a13 tle13
13
13
9 Click13 Save13
10 To13 view13 the13 visualisaon13 click13 the13 Dashboard13 page13
Page of 30 71 Page of 30 71
13
11 Click13 the13 Add13 Visualiza0on13 icon13
13
12 Select13 the13 visiualizaon13 that13 you13 just13 created13
13
Page of 31 71 Page of 31 71
13
13 To13 save13 the13 dashboard13 click13 the13 Save13 Dashboard13 icon13
Page of 32 71 Page of 32 71
13
14 Add13 a13 tle13 and13 select13 whether13 to13 store13 the13 selected13 me13 frame13 with13 the13 dashboard13
13
15 Click13 Save13
16 You13 can13 edit13 the13 shape13 and13 number13 of13 charts13 and13 save13 the13 dashboard13
17 To13 re-shy‐run13 the13 dashboard13 select13 the13 Dashboard13 page13 and13 click13 the13 Load13 Saved13 Dashboard13 icon13
Page of 33 71 Page of 33 71
13
18 Select13 the13 saved13 dashboard13 from13 the13 list13
13
19 The13 dashboard13 runs13
13
Page of 34 71 Page of 34 71
Page of 35 71 Page of 35 71
Aler0ng13 with13 ELK13 ELK13 examines13 the13 log13 records13 in13 process13 to13 send13 alerts13 to13 third-shy‐party13 alerng13 event13 management13 and13 social13 media13 applicaons13 As13 shown13 previously13 in13 the13 ELK13 architecture13 the13 ELK13 strategy13 for13 alerng13 is13 to13 send13 alerts13 from13 the13 Logstash13 component13 directly13 Therefore13 log13 records13 and13 alerts13 are13 sent13 to13 both13 Elascsearch13 and13 third-shy‐party13 tools13 at13 the13 same13 instant13 using13 the13 output13 secon13 of13 the13 Logstash13 configuraon13 Any13 alerts13 are13 generated13 during13 the13 filter13 secon13 processing13 of13 the13 Logstash13 configuraon13
Create13 alerts13 in13 Logstash13 The13 Logstash13 filter13 secon13 is13 used13 to13 trigger13 alerts13 based13 on13 the13 log13 records13 being13 processed13 For13 example13 log13 records13 can13 be13 checked13 for13 errors13 warnings13 crashed13 or13 stopped13 messages13 and13 generate13 an13 alert13
Here13 is13 an13 example13 Logstash13 filter13 code13
13 if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
The13 specific13 tags13 are13 set13 against13 the13 log13 records13 to13 trigger13 the13 alert13 later13 in13 the13 configuraon13 in13 the13 output13 secon13
Control13 the13 number13 of13 generated13 alerts13 using13 the13 Logstash13 throcle13 plugin13 so13 that13 either13 a13 lot13 of13 instances13 in13 a13 fixed13 me13 period13 are13 required13 before13 an13 alert13 is13 sent13 or13 a13 specific13 number13 of13 instances13 triggers13 the13 alert13
Page of 36 71 Page of 36 71
Edit13 the13 plugin13 parameters13 to13 meet13 system13 requirements13
13 13 13 13 13 if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 10
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle13
after_count =gt 20
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 30
period =gt 1200
key =gt servicename_alarm_warning
Page of 37 71 Page of 37 71
add_tag =gt servicename_throttled
13 13 13 13 13 13 13 13 13
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 40
period =gt 1500
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
Enrich13 alerts13 with13 message13 detail13 Some13 of13 the13 third-shy‐party13 integraons13 benefit13 from13 having13 components13 of13 the13 raw13 log13 record13 sent13 in13 addion13 to13 the13 raw13 record13 or13 fields13 in13 the13 record13 edited13 transformed13 or13 translated13
Grok13 Logstash13 allows13 the13 log13 record13 to13 be13 broken13 up13 into13 logical13 secons13 and13 each13 secon13 stored13 in13 a13 new13 parameter13 using13 the13 grok13 parameter13 in13 the13 filter13 secon13 Here13 is13 an13 example13 grok13
13 grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
To13 build13 the13 grok13 statement13 Logstash13 offers13 an13 online13 tool13 at13 hcpsgrokdebugherokuappcom13
Translate13 If13 specific13 fields13 in13 the13 log13 record13 are13 coded13 (for13 example13 the13 Bluemix13 app13 name13 is13 oaen13 not13 sent13 in13 the13 syslog13 messages13 instead13 the13 GUID13 is13 sent13 which13 is13 not13 useful)13 then13 the13 filter13 secon13 translate13 plugin13 can13 be13 used13 to13 add13 new13 fields13 by13 holding13 the13 decoded13 versions13 of13 those13 values13
13 translate
field =gt data1
destination =gt data1
override =gt true
Page of 38 71 Page of 38 71
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
dc50f6e4-c052-436a-bf27-95ba392c4228 monverifyservicemanagement
0c677e82-5bfc-4e38-9155-e416249e3ee6 ms-catalog-postpyretic-pomiculture
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
However13 the13 translate13 plugin13 is13 not13 included13 with13 the13 Logstash13 installaon13 In13 the13 next13 secon13 I13 will13 13 describe13 how13 to13 add13 new13 plugins13 to13 the13 Logstash13 instance13
Add13 custom13 plugins13 to13 Logstash13 As13 an13 open13 source13 tool13 Logstash13 encourages13 users13 to13 develop13 content13 for13 all13 users13 to13 share13 Some13 of13 the13 plugins13 that13 were13 created13 by13 our13 community13 include13 a13 translaon13 plugin13 an13 output13 Slack13 plugin13 and13 an13 output13 syslog13 plugin13 Letrsquos13 look13 at13 how13 to13 add13 these13 plugins13 to13 your13 environment13
Filter13 translate13 plugin13 To13 add13 the13 translate13 plugin13
1 Download13 the13 plugin13
2 Copy13 the13 plugin13 file13 logstash-shy‐filter-shy‐translate-shy‐214gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-filter-translate 214
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Output13 Slack13 plugin13 To13 add13 the13 Slack13 plugin13
Page of 39 71 Page of 39 71
1 Download13 the13 plugin13 13
2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐slack-shy‐014gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-output-slack 014
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Output13 syslog13 plugin13 To13 add13 the13 syslog13 plugin13
1 Download13 the13 plugin13 13
2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐syslog-shy‐214gem13 to13 the13 ELK13 Virtual13 Server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-output-syslog 214
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Integrate13 Logstash13 with13 Slack13 To13 send13 generated13 alerts13 to13 Slack13 complete13 these13 steps13
1 Create13 a13 Slack13 integraon13 13
2 Use13 the13 Slack13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 Slack13 integraon13
113 Login13 to13 Slack13 and13 go13 to13 the13 Apps13 page13 13
Page of 40 71 Page of 40 71
13
313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13
13
A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13
13
Page of 42 71 Page of 42 71
The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13
13
The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13
13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
slack
url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5
channel =gt cloud-operations-mon
username =gt nick_cawood
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13
Page of 43 71 Page of 43 71
Page of 44 71 Page of 44 71
In13 Slack13 the13 alerts13 look13 like13 this13
13
Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13
1 Create13 a13 PagerDuty13 integraon13 13
2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13
1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13
13
2 Click13 Add13 New13 Service13
13
313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13
Page of 45 71 Page of 45 71
13
413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13
513 Click13 Add13 Service13
13
The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13
Page of 46 71 Page of 46 71
13
The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt be7eb1ea9c224018923e818d3e891d0b
incident_key =gt logstashhosttype
Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13
Page of 47 71 Page of 47 71
13
Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13
1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13
2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13
3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13
4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13
13 13 http
url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo
http_method =gt post
format =gt json
For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13
13 13 syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
Page of 48 71 Page of 48 71
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13
13
The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13
if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2
This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13
Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13
Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13
Logstash13 severity Shown13 in13 Omnibus13 as
alert Indeterminate
crical Warning
debug 7
emergency Clear
error Minor
informaonal 6
noce Crical
warning Major
Page of 49 71 Page of 49 71
13
Page of 50 71 Page of 50 71
Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13
input
tcp
port =gt 5000
type =gt syslog
filter
mutate
add_field =gt format =gt cf
grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
translate
field =gt data1
destination =gt data1
override =gt true
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
=== Check Alarm Criteria ===
Standard Key Words
if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
=== Alarm Exclusions ===
if [message] =~ (i)sometexthere
mutate remove_tag =gt servicename_alarm
Throttle alarms so we do not hit PD rate limits
Page of 52 71 Page of 52 71
if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 5
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle
after_count =gt 5
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 5
period =gt 1200
key =gt servicename_alarm_warning
add_tag =gt servicename_throttled
Page of 53 71 Page of 53 71
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
output
elasticsearch
hosts =gt localhost
stdout codec =gt rubydebug
if servicename_alarm_err in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
Page of 54 71 Page of 54 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 55 71 Page of 55 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
Page of 56 71 Page of 56 71
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert err message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_error in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
Page of 57 71 Page of 57 71
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 58 71 Page of 58 71
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 59 71 Page of 59 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert error message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
Page of 60 71 Page of 60 71
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
Page of 61 71 Page of 61 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 62 71 Page of 62 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt x
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert fatal message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
Page of 63 71 Page of 63 71
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
Page of 64 71 Page of 64 71
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 65 71 Page of 65 71
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt heavy_exclamation_mark
Page of 66 71 Page of 66 71
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert WARNING message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 67 71 Page of 67 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
Page of 68 71 Page of 68 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 69 71 Page of 69 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
Page of 70 71 Page of 70 71
Page of 71 71 Page of 71 71
secons13 a13 variety13 of13 methods13 are13 available13 to13 process13 the13 data13 sent13 from13 the13 filter13 secon13 For13 example13 the13 records13 can13 be13 sent13 to13 external13 alerng13 tools13 such13 as13 PagerDuty13 social13 media13 tools13 such13 as13 Slack13 or13 event13 management13 hubs13 like13 IBM13 Netcool13 Omnibus13 via13 output13 methods13 syslog13 or13 hcp13 13
As13 an13 open13 source13 tool13 non-shy‐Logstash13 employees13 develop13 other13 methods13 and13 share13 them13 with13 the13 user13 community13 for13 addion13 to13 the13 library13 of13 methods13 Logstash13 can13 execute13 (examples13 of13 this13 process13 are13 included13 later13 in13 this13 document)13
Elas0csearch13 Elascsearch13 is13 the13 index13 store13 and13 query13 applicaon13 on13 the13 ELK13 stack13 It13 provides13 a13 standard13 full13 text13 search13 facility13 of13 the13 log13 file13 records13 processed13 by13 Logstash13 The13 ELK13 online13 help13 documentaon13 describes13 Elascsearch13 as13 follows13 ldquoIt13 is13 generally13 used13 as13 the13 underlying13 enginetechnology13 that13 powers13 applicaons13 that13 have13 complex13 search13 features13 and13 requirementsrdquo13
Kibana13 Kibana13 is13 the13 user13 interface13 for13 the13 ELK13 stack13 Kibana13 integrates13 with13 Elascsearch13 to13 visualise13 the13 results13 of13 searches13 and13 allow13 the13 creaon13 of13 charts13 and13 dashboards13 based13 on13 the13 searches13 The13 searches13 charts13 and13 dashboard13 can13 visualise13 real-shy‐me13 or13 historical13 log13 records13 Users13 access13 Kibana13 through13 an13 hcp13 URL13 from13 a13 client13 web13 browser13
ELK13 System13 Architecture13 The13 ELK13 stack13 can13 be13 described13 as13 follows13
Page of 4 71 Page of 4 71
13
Page of 5 71 Page of 5 71
Service13 Management13 architecture13 The13 IBM13 Cloud13 Architecture13 Center13 created13 a13 standard13 reference13 architecture13 for13 how13 to13 set13 up13 your13 cloud13 environemnt13 to13 handle13 incident13 management13 13
View13 the13 standard13 Service13 Management13 architecure13
13 13
Log13 Monitoring13 System13 architecture13 The13 following13 diagram13 shows13 a13 standard13 architecture13 for13 the13 log-shy‐monitoring13 system13 described13 in13 this13 tutorial13 13
13
Bluemix13 environment13 system13 architecture13 The13 ELK13 stack13 can13 be13 installed13 into13 a13 variety13 of13 environments13 When13 using13 Bluemix13 the13 following13 figure13 shows13 what13 the13 environment13 might13 look13 like13
Page of 6 71 Page of 6 71
13
The13 Bluemix13 cloud13 plaRorm13 offers13 a13 Public13 network13 and13 access13 to13 US13 South13 and13 UK13 environments13 From13 all13 these13 environments13 third-shy‐party13 cloud-shy‐based13 soluons13 are13 also13 accessible13
The13 ELK13 stack13 is13 installed13 in13 the13 Bluemix13 United13 Kingdom13 environment13 because13 this13 is13 the13 environment13 where13 Bluemix13 has13 virtual13 servers13 available13 for13 provisioning13 Other13 applicaons13 in13 the13 CASE13 project13 were13 also13 installed13 on13 virtual13 servers13 in13 the13 Bluemix13 United13 Kingdom13 region13
Bluemix13 apps13 were13 used13 to13 create13 a13 system13 to13 be13 monitored13 by13 ELK13 These13 apps13 were13 hosted13 in13 the13 Bluemix13 US13 South13 environment13 using13 Bluemix13 microservices13 which13 produce13 logs13 during13 normal13 operaon13 and13 during13 crashes13 and13 outages13
Through13 the13 Bluemix13 Cloud13 Foundry13 process13 the13 log13 records13 can13 be13 broadcast13 from13 the13 Bluemix13 apps13 so13 Logstash13 can13 receive13 them13 and13 send13 them13 to13 Elascsearch13 within13 the13 ELK13 stack13 or13 other13 virtual13 servers13 (hosng13 other13 IBM13 or13 third-shy‐party13 products)13 in13 the13 Bluemix13 United13 Kingdom13 environment13 or13 to13 third-shy‐party13 applicaons13 in13 the13 public13 cloud13 (Pager13 Duty13 Slack13 etc)13
Crea0ng13 a13 virtual13 server13 in13 Bluemix13 With13 Bluemix13 you13 can13 provision13 a13 virtual13 server13 in13 the13 cloud13 to13 install13 standalone13 or13 integrated13 system13 components13 Not13 all13 Bluemix13 regions13 offer13 this13 service13 all13 the13 me13 due13 to13 beta-shy‐project13 constraints13 In13 this13 arclersquos13 examples13 virtual13 servers13 were13 allowed13 to13 be13 provisioned13 in13 the13 United13 Kingdom13 region13 For13 more13 informaon13 read13 how13 to13 provision13 a13 Bluemix13 virtual13 server13 including13 some13 simple13 setup13 steps13 for13 ease13 of13 use13
Page of 7 71 Page of 7 71
Create13 an13 SSH13 key13 To13 access13 your13 virtual13 server13 once13 it13 is13 provisioned13 allocate13 an13 SSH13 key13 to13 the13 virtual13 server13
To13 generate13 a13 publicprivate13 RSA13 key13 pair13 follow13 these13 steps13
1 In13 a13 UNIX13 or13 Linuxreg13 window13 run13 the13 following13 command13 specifying13 a13 ltuniquekeynamegt13 of13 your13 your13 choice13 and13 a13 password13 or13 passphrase13 (memorize13 the13 password)$13 ssh-shy‐keygen13 -shy‐t13 rsa13 -shy‐f13 ltuniquekeynamegt13
2 Enter13 the13 passphrase13 (empty13 for13 no13 passphrase)13
3 Enter13 the13 same13 passphrase13 again13
Your13 idenficaon13 has13 been13 saved13 in13 ltuniquekeynamegt13
Your13 public13 key13 has13 been13 saved13 in13 ltuniquekeynamegtpub13
The13 key13 fingerprint13 is13
d6268464506525930ce4632c693c203c13 ltusergtltservergt13
The13 keys13 randomart13 image13 is13
+--[ RSA 2048]----+
|o ++=o |
| E o ooo |
| |
| + o |
| S o |
| o |
| |
| |
| |
+-----------------+
Two13 keys13 are13 created13 public13 and13 private13 The13 public13 key13 contents13 are13 used13 to13 create13 a13 virtual13 server13 with13 a13 public13 IP13 address13 The13 private13 key13 is13 used13 to13 access13 the13 virtual13 server13 from13 a13 client13 machine13
Page of 8 71 Page of 8 71
Provisioning13 a13 virtual13 server13 1 A13 Bluemix13 account13 is13 required13 for13 this13 process13 Sign13 up13 here13 and13 log13 in13 Make13 sure13 your13
environment13 is13 correct13 For13 example13 if13 you13 are13 in13 the13 United13 Kingdom13 select13 the13 United13 Kingdom13 region13 in13 the13 Bluemix13 console13
13
13
2 Add13 a13 virtual13 server13 to13 your13 Bluemix13 environment13 Click13 Run13 Virtual13 Servers13
13
3 Select13 an13 OS13 image13 from13 the13 EXISTING13 list13 For13 ELK13 select13 Centos13 OS13 (use13 latest13 available13 version)13
Page of 9 71 Page of 9 71
4 Name13 the13 virtual13 server13
5 Make13 sure13 Assign13 public13 IP13 address13 is13 checked13
13
6 Click13 Add13 Key13 to13 add13 the13 SSH13 key13 you13 created13 previously13 Copy13 the13 contents13 of13 the13 public13 key13 into13 the13 Public13 Key13 to13 import13 field13
Page of 10 71 Page of 10 71
13
7 Click13 OK13 Then13 click13 CREATE13
The13 virtual13 server13 is13 provisioned13 and13 added13 to13 the13 Bluemix13 dashboard13 under13 VIRTUAL13 SERVERS13
Page of 11 71 Page of 11 71
13
Set13 up13 the13 virtual13 server13
Log13 on13 to13 the13 virtual13 server13 Now13 you13 can13 log13 on13 to13 the13 virtual13 server13 and13 add13 new13 users13 and13 programs13 to13 make13 it13 easier13 to13 access13 the13 virtual13 server13
To13 access13 the13 server13 go13 to13 the13 UNIX13 or13 Linux13 window13 that13 you13 used13 to13 create13 the13 SSH13 key13 In13 the13 directory13 where13 the13 ltuniquekeynamegt13 files13 exist13 run13 the13 following13 command13 referencing13 the13 SSH13 keyname13 the13 public13 IP13 address13 Bluemix13 allocated13 for13 the13 virtual13 server13 and13 the13 password13 for13 the13 SSH13 key13
[userserver ~]$ ssh -i ltuniquekeynamegt ibmcloudltpublicIPaddressgt
Enter13 the13 passphrase13 for13 the13 key13 ltuniquekeynamegt13
Last login Tue Jul 5 125546 2016 from 108-210-96-231lightspeedrlghncsbcglobalnet
[ibmcloudelkforcase ~]$
ibmcloud13 is13 the13 default13 user13 for13 the13 Bluemix13 virtual13 servers13 and13 allows13 SSH13 to13 access13 the13 server13
Acces13 the13 root13 user13 To13 access13 the13 root13 user13 log13 on13 as13 ibmcloud13 user13 and13 run13 the13 following13 command13
[ibmcloudelkforcase ~]$ sudo bash
[rootelkforcase ibmcloud]
The13 root13 user13 can13 be13 used13 to13 add13 programs13 to13 the13 server13 and13 create13 and13 edit13 users13 as13 required13 to13 access13 applicaons13 installed13 on13 the13 virtual13 server13
Page of 12 71 Page of 12 71
Copy13 files13 to13 the13 virtual13 server13 To13 move13 installaon13 files13 to13 the13 server13 go13 to13 the13 UNIX13 or13 Linux13 window13 you13 used13 to13 create13 the13 SSH13 key13 and13 the13 directory13 where13 the13 ltuniquekeynamegt13 files13 exist13 Move13 the13 installaon13 files13 to13 the13 same13 locaon13
Run13 the13 following13 command13 referencing13 the13 SSH13 keyname13 the13 public13 IP13 address13 Bluemix13 allocated13 for13 the13 virtual13 server13 and13 the13 password13 for13 the13 SSH13 key13
[userserver ~]$ scp ndashi ltuniquekeynamegt ltINSTALLATIONFILENAMEgt ibmcloudltpublicIPaddressgthomeibmcloud
Enter13 the13 passphrase13 for13 key13 cloudkey313
This13 copies13 the13 file13 using13 the13 ibmcloud13 user13 to13 the13 homeibmcloud13 directory13 on13 the13 virtual13 server13
Install13 and13 access13 the13 virtual13 server13 via13 VNC13 server13 Using13 the13 ssh13 command13 to13 access13 the13 virtual13 server13 is13 not13 the13 most13 stable13 method13 and13 does13 not13 offer13 X-shy‐Windows13 support13 Therefore13 we13 suggest13 that13 you13 use13 a13 VNC13 client13 to13 access13 the13 virtual13 server13
Installing13 a13 VNC13 client13 should13 be13 done13 as13 the13 root13 user13 Use13 the13 yum13 command13 as13 follows13
[ibmcloudelkforcase ~]$ yum groupinstall GNOME Desktop
[ibmcloudelkforcase ~]$ yum install tigervnc-server
These13 commands13 first13 install13 a13 desktop13 style13 OS13 user13 interface13 and13 then13 a13 VNC13 client13 to13 remotely13 access13 the13 desktop13 style13 OS13
Then13 as13 either13 the13 root13 user13 or13 another13 user13 (example13 below13 uses13 elk)13 run13 the13 following13 command13 to13 create13 the13 VNC13 client13 connecon13
[elkelkforcase ~]$ vncserver
New elkforcase2 (elk) desktop is elkforcase2
Starting applications specified in homeelkvncxstartup
Log file is homeelkvncelkforcase2log
[elkelkforcase ~]$
Install13 a13 VNC13 client13 on13 the13 client13 machine13 used13 to13 access13 the13 Internet13 and13 (in13 the13 example13 above)13 use13 the13 command13 ltpublicIPadddressgt213 and13 the13 password13 for13 ELK13 to13 access13 the13 virtual13 server13
Using13 a13 VNC13 client13 means13 that13 you13 can13 leave13 long13 duraon13 installaons13 running13 without13 fear13 of13 the13 terminal13 losing13 connecon13 and13 the13 installaons13 failing13
Install13 the13 ELK13 stack13 Download13 the13 source13 soaware13 code13 for13 the13 ELK13 stack13 from13 the13 ELK13 website13 13
At13 the13 me13 of13 wring13 the13 downloaded13 soaware13 files13 were13
Page of 13 71 Page of 13 71
13
Transfer13 the13 files13 to13 the13 virtual13 server13 (using13 the13 scp13 method13 detailed13 above)13
Install13 Elas0csearch13 As13 a13 root13 user13 create13 a13 non-shy‐root13 user13 with13 the13 adduser13 command13
Create13 a13 directory13 to13 install13 Elascsearch13 into13 and13 untar13 the13 contents13 of13 the13 Elascsearch13 install13 file13 into13 that13 directory13
[elkelkforcase]$ cd home
[elkelkforcase]$ mkdir elasticsearch
[elkelkforcase]$ cp homeibmcloudelasticsearch-233targz homeelasticsearch
[elkelkforcase]$ cd elasticsearch
[elkelkforcase]$ gunzip elasticsearch-233targz
[elkelkforcase]$ untar elasticsearch-233tar
Elascsearch13 is13 now13 installed13
Start13 Elas0csearch13 To13 start13 Elascsearch13 go13 to13 the13 bin13 directory13 as13 the13 non-shy‐root13 user13 when13 you13 installed13 it13 and13 run13 the13 elascsearch13 executable13
[elkelkforcase]$ cd homeelasticsearchelasticsearch-233bin
[elkelkforcase]$ elasticsearch amp
[2] 22211
[elkelkforcase]$ [2016-07-05 133857730][INFO ][node] [Arsenal] version[233] pid[22211] build[218bdf12016-05-17T154004Z]
[2016-07-05 133857740][INFO ][node] [Arsenal] initializing
[2016-07-05 133858984][INFO ][plugins] [Arsenal] modules [reindex lang-expression lang-groovy] plugins [] sites []
[2016-07-05 133859146][INFO ][env] [Arsenal] using [1] data paths mounts [[ (rootfs)]] net usable_space [155gb] net total_space [199gb] spins [unknown] types [rootfs]
[2016-07-05 133859146][INFO ][env] [Arsenal] heap size [10156mb] compressed ordinary object pointers [true]
Page of 14 71 Page of 14 71
[2016-07-05 133859147][WARN ][env] [Arsenal] max file descriptors [4096] for elasticsearch process likely too low consider increasing to at least [65536]
[2016-07-05 133903526][INFO ][node] [Arsenal] initialized
[2016-07-05 133903526][INFO ][node] [Arsenal] starting
To13 test13 that13 Elascsearch13 is13 running13 correctly13 open13 the13 browser13 in13 the13 VNC13 client13 and13 run13 localhost920013
13
Install13 Kibana13 ELK13 recommends13 installing13 and13 running13 Kibana13 as13 root13 13
1 As13 root13 user13 create13 an13 installaon13 directory13 and13 untar13 the13 installaon13 file13
[rootelkforcase]$ cd home
[rootelkforcase]$ mkdir kibana
[rootelkforcase]$ cp homeibmcloudkibana-451-linux-x64targz homekibana
[rootelkforcase]$ cd kibana
[rootelkforcase]$ gunzip kibana-451-linux-x64targz
[rootelkforcase]$ untar kibana-451-linux-x64tar
2 Modify13 the13 manifestyml13 configuraon13 file13 Go13 to13 the13 kibana-shy‐451-shy‐linux-shy‐x6413 directory13
[rootelkforcase]$ cd homekibanakibana-451-linux-x64
3 Edit13 the13 manifestyml13 file13 so13 the13 contents13 are13
applicaons13
-shy‐ name13 kibana-shy‐in-shy‐bluemix13
memory13 128M13
host13 kibana-shy‐in-shy‐bluemix13
buildpack13 hcpsgithubcomcloudfoundry-shy‐communitynginx-shy‐buildpackgit13
Page of 15 71 Page of 15 71
4 Edit13 the13 configkibanayml13 file13 so13 the13 contents13 are13
Kibana is served by a back end server This controls which port to use
port 5601
The host to bind the server to
host 0000
If you are running kibana behind a proxy and want to mount it at a path
specify that path here The basePath cant end in a slash
serverbasePath
The maximum payload size in bytes on incoming server requests
servermaxPayloadBytes 1048576
The Elasticsearch instance to use for all your queries
elasticsearchurl httplocalhost9200
preserve_elasticsearch_host true will send the hostname specified in `elasticsearch` If you set it to false
then the host you use to connect to this Kibana instance will be sent
elasticsearchpreserveHost true
Kibana uses an index in Elasticsearch to store saved searches visualizations
and dashboards It will create a new index if it doesnt already exist
kibanaindex kibana
The default application to load
kibanadefaultAppId discover
If your Elasticsearch is protected with basic auth these are the user credentials
used by the Kibana server to perform maintenance on the kibana_index at startup Your
Kibana users will still need to authenticate with Elasticsearch (which is proxied through
the Kibana server)
Page of 16 71 Page of 16 71
elasticsearchusername user
elasticsearchpassword pass
SSL for outgoing requests from the Kibana Server to the browser (PEM formatted)
serversslcert pathtoyourservercrt
serversslkey pathtoyourserverkey
Optional setting to validate that your Elasticsearch backend uses the same key files (PEM formatted)
elasticsearchsslcert pathtoyourclientcrt
elasticsearchsslkey pathtoyourclientkey
If you need to provide a CA certificate for your Elasticsearch instance put
the path of the pem file here
elasticsearchsslca pathtoyourCApem
Set to false to have a complete disregard for the validity of the SSL
certificate
elasticsearchsslverify true
Time in milliseconds to wait for elasticsearch to respond to pings defaults to
request_timeout setting
elasticsearchpingTimeout 1500
Time in milliseconds to wait for responses from the back end or elasticsearch
This must be gt 0
elasticsearchrequestTimeout 30000
Time in milliseconds for Elasticsearch to wait for responses from shards
Set to 0 to disable
elasticsearchshardTimeout 0
Time in milliseconds to wait for Elasticsearch at Kibana startup before retrying
elasticsearchstartupTimeout 5000
elasticsearchstartupTimeout 30000
Page of 17 71 Page of 17 71
Set the path to where you would like the process id file to be created
pidfile varrunkibanapid
If you would like to send the log output to a file you can set the path below
loggingdest stdout
Set this to true to suppress all logging output
loggingsilent false
Set this to true to suppress all logging output except for error messages
loggingquiet false
Set this to true to log all events including system usage information and all requests
loggingverbose false
Kibana13 is13 now13 installed13
Start13 Kibana13 To13 start13 Kibana13 go13 to13 the13 bin13 directory13 and13 run13 the13 kibana13 executable13
[rootelkforcase]$
[rootelkforcase]$ cd homekibanakibana-451-linux-x64bin
[rootelkforcase]$ kibana amp
[1] 22283
[rootelkforcase] log [095900542] [info][status][pluginkibana] Status changed from uninitialized to green - Ready
log [095900600] [info][status][pluginelasticsearch] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [095900689] [info][status][pluginkbn_vislib_vis_types] Status changed from uninitialized to green - Ready
log [095900717] [info][status][pluginmarkdown_vis] Status changed from uninitialized to green - Ready
log [095900721] [info][status][pluginmetric_vis] Status changed from uninitialized to green - Ready
log [095900814] [info][status][pluginspyModes] Status changed from uninitialized to green - Ready
log [095900934] [info][status][pluginstatusPage] Status changed from uninitialized to green - Ready
Page of 18 71 Page of 18 71
log [095900941] [info][status][plugintable_vis] Status changed from uninitialized to green - Ready
log [095901067] [info][listening] Server running at http00005601
log [095906083] [info][status][pluginelasticsearch] Status changed from yellow to yellow - No existing Kibana index found
log [095908942] [info][status][pluginelasticsearch] Status changed from yellow to green - Kibana index ready
Kibana13 is13 now13 started13
Broadcast13 Bluemix13 app13 log13 records13 Bluemix13 apps13 (which13 are13 shown13 in13 the13 US13 South13 Bluemix13 region13 of13 the13 System13 Architecture13 diagram)13 will13 produce13 logs13 in13 a13 syslog13 rfc542413 format13 using13 a13 Bluemix13 tool13 called13 Loggregator13
For13 Logstash13 to13 receive13 these13 logs13 each13 Bluemix13 app13 needs13 to13 broadcast13 these13 logs13 To13 do13 that13 run13 the13 following13 commands13 on13 each13 Bluemix13 app13 using13 the13 public13 IP13 address13 of13 the13 ELK13 virtual13 server13 the13 ELK13 tag13 to13 trace13 the13 process13 and13 insert13 the13 name13 of13 each13 app13 in13 turn13
cf cups elk -l syslogltpublicIPaddress5000 cf bind-service ltmyappgt elk
cf restage ltmyappgt
Install13 Logstash13 To13 install13 Logstash13 as13 root13 user13 create13 a13 directory13 for13 the13 installaon13 and13 untar13 the13 installaon13 file13 there13
13 13 [rootelkforcase]$ cd home
[rootelkforcase]$ mkdir logstash
[rootelkforcase]$ cp homeibmcloudlogstash-233targz homelogstash
[rootelkforcase]$ cd logstash
[rootelkforcase]$ gunzip logstash-233targz
[rootelkforcase]$ untar logstash-233tar
Write13 Logstash13 configura0on13 Logstash13 requires13 a13 configuraon13 file13 to13 be13 wricen13 to13 receive13 transform13 and13 output13 the13 log13 records13 to13 Elascsearch13 This13 secon13 explains13 the13 basic13 requirements13 of13 processing13 Bluemix13 app13 log13 records13 into13 Elascsearch13 (more13 complex13 configuraons13 are13 detailed13 later13 in13 this13 document13 for13 the13 integraons13 with13 third-shy‐party13 tools)13
Create13 a13 conf13 file13 as13 root13 user13
[rootelkforcase]$ cd homelogstashlogstash-233
[rootelkforcase]$ mkdir config
Page of 19 71 Page of 19 71
[rootelkforcase]$ cd config
[rootelkforcase]$ touch loggregatorconf
Edit13 the13 loggregator13 file13 with13 the13 following13 contents13
13 Input13 secon13 defines13 which13 log13 records13 to13 receive13
13 port13 and13 type13 reflect13 method13 selected13 for13 Cloud13 Foundry13 to13 broadcast13 the13 records13
input
tcp
port =gt 5000
type =gt syslog
13 Filter13 secon13 ndash13 transforms13 the13 data13
filter
Add a field to identify the records as Cloud Foundry based
mutate
add_field =gt format =gt cf
13 Output13 secon13 ndash13 sends13 the13 data13 to13 its13 desnaon(s)13
output
13 Use13 the13 elascsearch13 plugin13 to13 send13 the13 data13 to13 the13 elascsearch13 instance13
13 elasticsearch
hosts =gt localhost
stdout codec =gt rubydebug
13 End13 of13 Config13
Save13 the13 file13 Verify13 the13 format13 of13 the13 file13 is13 acceptable13 by13 running13 the13 following13 commands13
[rootelkforcase]$ cd homelogstashlogstash-233
[rootelkforcase]$ binlogstash agent --configtest -f configloggregatorconf
Page of 20 71 Page of 20 71
Configuraon13 is13 OK13
Start13 Logstash13 Aaer13 you13 have13 wricen13 and13 verified13 the13 Logstash13 configuraon13 you13 can13 start13 Logstash13 to13 process13 records13 from13 the13 Bluemix13 apps13 (via13 Cloud13 Foundry)13 Run13 the13 following13 as13 root13 user13
[rootelkforcase]$ cd homelogstashlogstash-233
[rootelkforcase]$ binlogstash agent -f configloggregatorconf
Logstash13 is13 now13 started13
Monitor13 Logstash13 processing13 Once13 Logstash13 is13 running13 and13 log13 records13 are13 being13 produced13 by13 the13 Bluemix13 apps13 you13 can13 monitor13 progress13 in13 the13 same13 window13 that13 was13 used13 to13 start13 Logstash13 Output13 is13 similar13 to13 the13 following13
13
The13 log13 records13 being13 broadcast13 from13 the13 Bluemix13 apps13 are13 now13 sent13 to13 Elascsearch13 from13 the13 Logstash13 output13 configuraon13 Records13 are13 sent13 in13 real13 me13 If13 Logstash13 is13 not13 running13 while13 log13 records13 are13 being13 produced13 then13 these13 records13 will13 not13 be13 processed13 by13 Logstash13 and13 will13 not13 be13 stored13 in13 Elascsearch13
Page of 21 71 Page of 21 71
Use13 ELK13 The13 log13 records13 from13 the13 Bluemix13 apps13 are13 now13 being13 stored13 and13 indexed13 by13 the13 Elascsearch13 component13 This13 data13 can13 be13 accessed13 via13 the13 Kibana13 applicaon13
Kibana13 Discover13 page13 Using13 a13 web13 browser13 either13 directly13 from13 a13 client13 or13 using13 the13 VNC13 client13 access13 Kibana13 using13 the13 publicIPaddress13 of13 the13 ELK13 virtual13 server13
hcpltpublicIPaddressgt560113
No13 userpassword13 credenals13 are13 required13
Kibana13 defaults13 to13 a13 search13 of13 all13 records13 ()13 in13 the13 last13 1513 minutes13 and13 returns13 the13 results13
13
More13 complex13 searches13 are13 possible13 by13 changing13 the13 me13 filter13 and13 supplying13 search13 queries13
Change13 the13 0me13 filter13 To13 change13 the13 me13 ranges13 click13 the13 clock13 symbol13 in13 the13 top-shy‐right13 of13 the13 Kibana13 Discover13 page13
13
You13 will13 see13 the13 following13 opons13
13
Page of 22 71 Page of 22 71
A13 variety13 of13 quick13 me13 ranges13 can13 be13 selected13 or13 you13 can13 switch13 to13 relave13 or13 absolute13 me13 ranges13
Quick13 me13 ranges13 allow13 switching13 between13 last13 X13 minutes13 to13 last13 Y13 hours13 or13 Z13 days13 etc13
Relave13 me13 ranges13 allow13 more13 specific13 versions13 of13 the13 Quick13 opons13 from13 specific13 minuteshoursdays13 to13 now13
13
Absolute13 me13 ranges13 specify13 the13 start13 and13 end13 mes13 for13 the13 searches13
13
Custom13 search13 queries13 You13 can13 use13 the13 text13 window13 of13 the13 Discover13 page13 to13 search13 for13 matches13 within13 the13 log13 records13 You13 can13 use13 Apache13 Lucene13 or13 Elascsearch13 query13 DSL13 syntax13
13
Page of 23 71 Page of 23 71
Saved13 searches13 You13 can13 save13 any13 custom13 search13 syntax13 and13 choice13 of13 me13 range13 for13 future13 ease13 of13 use13 Any13 relave13 or13 absolute13 me13 frame13 is13 saved13 as13 part13 of13 the13 search13 13
To13 save13 a13 search13
1 Set13 up13 and13 run13 the13 required13 syntax13 and13 me13 frame13
13
2 Click13 the13 Save13 Search13 icon13
13
3 Type13 in13 a13 name13 for13 the13 saved13 search13
Page of 24 71 Page of 24 71
13
7 To13 delete13 or13 update13 saved13 searches13 click13 13 manage13 saved13 searches13
13
Use13 the13 edit13 icons13 to13 make13 any13 changes13
Page of 26 71 Page of 26 71
13
Build13 simple13 Kibana13 histograms13 Kibana13 allows13 you13 to13 build13 charts13 and13 dashboards13 based13 on13 the13 search13 capability13 offered13 by13 the13 Discover13 page13 Saved13 searches13 can13 be13 used13 as13 the13 basis13 for13 charts13 and13 dashboards13 built13 using13 those13 charts13 To13 build13 a13 simple13 histogram13
1 Select13 the13 Visualize13 page13
13
2 Select13 Ver0cal13 Bar13 Chart13
13
3 To13 use13 a13 saved13 search13 click13 From13 a13 saved13 search13
Page of 27 71 Page of 27 71
13
4 Select13 a13 previously13 saved13 search13
13
5 Select13 to13 edit13 the13 x-shy‐axis
6 Edit13 the13 fields13 with13 informaon13 about13 the13 aggregaon13 field13 interval13 and13 custom13 label13
Page of 28 71 Page of 28 71
13
Page of 29 71 Page of 29 71
7 Click13 the13 green13 arrow13 The13 histogram13 is13 created13 and13 run13 based13 on13 the13 saved13 search
13
8 To13 save13 the13 histogram13 click13 the13 Save13 Visualiza0on13 icon13 and13 add13 a13 tle13
13
13
9 Click13 Save13
10 To13 view13 the13 visualisaon13 click13 the13 Dashboard13 page13
Page of 30 71 Page of 30 71
13
11 Click13 the13 Add13 Visualiza0on13 icon13
13
12 Select13 the13 visiualizaon13 that13 you13 just13 created13
13
Page of 31 71 Page of 31 71
13
13 To13 save13 the13 dashboard13 click13 the13 Save13 Dashboard13 icon13
Page of 32 71 Page of 32 71
13
14 Add13 a13 tle13 and13 select13 whether13 to13 store13 the13 selected13 me13 frame13 with13 the13 dashboard13
13
15 Click13 Save13
16 You13 can13 edit13 the13 shape13 and13 number13 of13 charts13 and13 save13 the13 dashboard13
17 To13 re-shy‐run13 the13 dashboard13 select13 the13 Dashboard13 page13 and13 click13 the13 Load13 Saved13 Dashboard13 icon13
Page of 33 71 Page of 33 71
13
18 Select13 the13 saved13 dashboard13 from13 the13 list13
13
19 The13 dashboard13 runs13
13
Page of 34 71 Page of 34 71
Page of 35 71 Page of 35 71
Aler0ng13 with13 ELK13 ELK13 examines13 the13 log13 records13 in13 process13 to13 send13 alerts13 to13 third-shy‐party13 alerng13 event13 management13 and13 social13 media13 applicaons13 As13 shown13 previously13 in13 the13 ELK13 architecture13 the13 ELK13 strategy13 for13 alerng13 is13 to13 send13 alerts13 from13 the13 Logstash13 component13 directly13 Therefore13 log13 records13 and13 alerts13 are13 sent13 to13 both13 Elascsearch13 and13 third-shy‐party13 tools13 at13 the13 same13 instant13 using13 the13 output13 secon13 of13 the13 Logstash13 configuraon13 Any13 alerts13 are13 generated13 during13 the13 filter13 secon13 processing13 of13 the13 Logstash13 configuraon13
Create13 alerts13 in13 Logstash13 The13 Logstash13 filter13 secon13 is13 used13 to13 trigger13 alerts13 based13 on13 the13 log13 records13 being13 processed13 For13 example13 log13 records13 can13 be13 checked13 for13 errors13 warnings13 crashed13 or13 stopped13 messages13 and13 generate13 an13 alert13
Here13 is13 an13 example13 Logstash13 filter13 code13
13 if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
The13 specific13 tags13 are13 set13 against13 the13 log13 records13 to13 trigger13 the13 alert13 later13 in13 the13 configuraon13 in13 the13 output13 secon13
Control13 the13 number13 of13 generated13 alerts13 using13 the13 Logstash13 throcle13 plugin13 so13 that13 either13 a13 lot13 of13 instances13 in13 a13 fixed13 me13 period13 are13 required13 before13 an13 alert13 is13 sent13 or13 a13 specific13 number13 of13 instances13 triggers13 the13 alert13
Page of 36 71 Page of 36 71
Edit13 the13 plugin13 parameters13 to13 meet13 system13 requirements13
13 13 13 13 13 if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 10
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle13
after_count =gt 20
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 30
period =gt 1200
key =gt servicename_alarm_warning
Page of 37 71 Page of 37 71
add_tag =gt servicename_throttled
13 13 13 13 13 13 13 13 13
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 40
period =gt 1500
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
Enrich13 alerts13 with13 message13 detail13 Some13 of13 the13 third-shy‐party13 integraons13 benefit13 from13 having13 components13 of13 the13 raw13 log13 record13 sent13 in13 addion13 to13 the13 raw13 record13 or13 fields13 in13 the13 record13 edited13 transformed13 or13 translated13
Grok13 Logstash13 allows13 the13 log13 record13 to13 be13 broken13 up13 into13 logical13 secons13 and13 each13 secon13 stored13 in13 a13 new13 parameter13 using13 the13 grok13 parameter13 in13 the13 filter13 secon13 Here13 is13 an13 example13 grok13
13 grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
To13 build13 the13 grok13 statement13 Logstash13 offers13 an13 online13 tool13 at13 hcpsgrokdebugherokuappcom13
Translate13 If13 specific13 fields13 in13 the13 log13 record13 are13 coded13 (for13 example13 the13 Bluemix13 app13 name13 is13 oaen13 not13 sent13 in13 the13 syslog13 messages13 instead13 the13 GUID13 is13 sent13 which13 is13 not13 useful)13 then13 the13 filter13 secon13 translate13 plugin13 can13 be13 used13 to13 add13 new13 fields13 by13 holding13 the13 decoded13 versions13 of13 those13 values13
13 translate
field =gt data1
destination =gt data1
override =gt true
Page of 38 71 Page of 38 71
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
dc50f6e4-c052-436a-bf27-95ba392c4228 monverifyservicemanagement
0c677e82-5bfc-4e38-9155-e416249e3ee6 ms-catalog-postpyretic-pomiculture
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
However13 the13 translate13 plugin13 is13 not13 included13 with13 the13 Logstash13 installaon13 In13 the13 next13 secon13 I13 will13 13 describe13 how13 to13 add13 new13 plugins13 to13 the13 Logstash13 instance13
Add13 custom13 plugins13 to13 Logstash13 As13 an13 open13 source13 tool13 Logstash13 encourages13 users13 to13 develop13 content13 for13 all13 users13 to13 share13 Some13 of13 the13 plugins13 that13 were13 created13 by13 our13 community13 include13 a13 translaon13 plugin13 an13 output13 Slack13 plugin13 and13 an13 output13 syslog13 plugin13 Letrsquos13 look13 at13 how13 to13 add13 these13 plugins13 to13 your13 environment13
Filter13 translate13 plugin13 To13 add13 the13 translate13 plugin13
1 Download13 the13 plugin13
2 Copy13 the13 plugin13 file13 logstash-shy‐filter-shy‐translate-shy‐214gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-filter-translate 214
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Output13 Slack13 plugin13 To13 add13 the13 Slack13 plugin13
Page of 39 71 Page of 39 71
1 Download13 the13 plugin13 13
2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐slack-shy‐014gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-output-slack 014
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Output13 syslog13 plugin13 To13 add13 the13 syslog13 plugin13
1 Download13 the13 plugin13 13
2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐syslog-shy‐214gem13 to13 the13 ELK13 Virtual13 Server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-output-syslog 214
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Integrate13 Logstash13 with13 Slack13 To13 send13 generated13 alerts13 to13 Slack13 complete13 these13 steps13
1 Create13 a13 Slack13 integraon13 13
2 Use13 the13 Slack13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 Slack13 integraon13
113 Login13 to13 Slack13 and13 go13 to13 the13 Apps13 page13 13
Page of 40 71 Page of 40 71
13
313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13
13
A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13
13
Page of 42 71 Page of 42 71
The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13
13
The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13
13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
slack
url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5
channel =gt cloud-operations-mon
username =gt nick_cawood
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13
Page of 43 71 Page of 43 71
Page of 44 71 Page of 44 71
In13 Slack13 the13 alerts13 look13 like13 this13
13
Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13
1 Create13 a13 PagerDuty13 integraon13 13
2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13
1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13
13
2 Click13 Add13 New13 Service13
13
313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13
Page of 45 71 Page of 45 71
13
413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13
513 Click13 Add13 Service13
13
The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13
Page of 46 71 Page of 46 71
13
The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt be7eb1ea9c224018923e818d3e891d0b
incident_key =gt logstashhosttype
Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13
Page of 47 71 Page of 47 71
13
Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13
1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13
2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13
3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13
4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13
13 13 http
url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo
http_method =gt post
format =gt json
For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13
13 13 syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
Page of 48 71 Page of 48 71
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13
13
The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13
if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2
This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13
Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13
Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13
Logstash13 severity Shown13 in13 Omnibus13 as
alert Indeterminate
crical Warning
debug 7
emergency Clear
error Minor
informaonal 6
noce Crical
warning Major
Page of 49 71 Page of 49 71
13
Page of 50 71 Page of 50 71
Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13
input
tcp
port =gt 5000
type =gt syslog
filter
mutate
add_field =gt format =gt cf
grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
translate
field =gt data1
destination =gt data1
override =gt true
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
=== Check Alarm Criteria ===
Standard Key Words
if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
=== Alarm Exclusions ===
if [message] =~ (i)sometexthere
mutate remove_tag =gt servicename_alarm
Throttle alarms so we do not hit PD rate limits
Page of 52 71 Page of 52 71
if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 5
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle
after_count =gt 5
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 5
period =gt 1200
key =gt servicename_alarm_warning
add_tag =gt servicename_throttled
Page of 53 71 Page of 53 71
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
output
elasticsearch
hosts =gt localhost
stdout codec =gt rubydebug
if servicename_alarm_err in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
Page of 54 71 Page of 54 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 55 71 Page of 55 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
Page of 56 71 Page of 56 71
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert err message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_error in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
Page of 57 71 Page of 57 71
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 58 71 Page of 58 71
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 59 71 Page of 59 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert error message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
Page of 60 71 Page of 60 71
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
Page of 61 71 Page of 61 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 62 71 Page of 62 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt x
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert fatal message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
Page of 63 71 Page of 63 71
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
Page of 64 71 Page of 64 71
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 65 71 Page of 65 71
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt heavy_exclamation_mark
Page of 66 71 Page of 66 71
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert WARNING message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 67 71 Page of 67 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
Page of 68 71 Page of 68 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 69 71 Page of 69 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
Page of 70 71 Page of 70 71
Page of 71 71 Page of 71 71
13
Page of 5 71 Page of 5 71
Service13 Management13 architecture13 The13 IBM13 Cloud13 Architecture13 Center13 created13 a13 standard13 reference13 architecture13 for13 how13 to13 set13 up13 your13 cloud13 environemnt13 to13 handle13 incident13 management13 13
View13 the13 standard13 Service13 Management13 architecure13
13 13
Log13 Monitoring13 System13 architecture13 The13 following13 diagram13 shows13 a13 standard13 architecture13 for13 the13 log-shy‐monitoring13 system13 described13 in13 this13 tutorial13 13
13
Bluemix13 environment13 system13 architecture13 The13 ELK13 stack13 can13 be13 installed13 into13 a13 variety13 of13 environments13 When13 using13 Bluemix13 the13 following13 figure13 shows13 what13 the13 environment13 might13 look13 like13
Page of 6 71 Page of 6 71
13
The13 Bluemix13 cloud13 plaRorm13 offers13 a13 Public13 network13 and13 access13 to13 US13 South13 and13 UK13 environments13 From13 all13 these13 environments13 third-shy‐party13 cloud-shy‐based13 soluons13 are13 also13 accessible13
The13 ELK13 stack13 is13 installed13 in13 the13 Bluemix13 United13 Kingdom13 environment13 because13 this13 is13 the13 environment13 where13 Bluemix13 has13 virtual13 servers13 available13 for13 provisioning13 Other13 applicaons13 in13 the13 CASE13 project13 were13 also13 installed13 on13 virtual13 servers13 in13 the13 Bluemix13 United13 Kingdom13 region13
Bluemix13 apps13 were13 used13 to13 create13 a13 system13 to13 be13 monitored13 by13 ELK13 These13 apps13 were13 hosted13 in13 the13 Bluemix13 US13 South13 environment13 using13 Bluemix13 microservices13 which13 produce13 logs13 during13 normal13 operaon13 and13 during13 crashes13 and13 outages13
Through13 the13 Bluemix13 Cloud13 Foundry13 process13 the13 log13 records13 can13 be13 broadcast13 from13 the13 Bluemix13 apps13 so13 Logstash13 can13 receive13 them13 and13 send13 them13 to13 Elascsearch13 within13 the13 ELK13 stack13 or13 other13 virtual13 servers13 (hosng13 other13 IBM13 or13 third-shy‐party13 products)13 in13 the13 Bluemix13 United13 Kingdom13 environment13 or13 to13 third-shy‐party13 applicaons13 in13 the13 public13 cloud13 (Pager13 Duty13 Slack13 etc)13
Crea0ng13 a13 virtual13 server13 in13 Bluemix13 With13 Bluemix13 you13 can13 provision13 a13 virtual13 server13 in13 the13 cloud13 to13 install13 standalone13 or13 integrated13 system13 components13 Not13 all13 Bluemix13 regions13 offer13 this13 service13 all13 the13 me13 due13 to13 beta-shy‐project13 constraints13 In13 this13 arclersquos13 examples13 virtual13 servers13 were13 allowed13 to13 be13 provisioned13 in13 the13 United13 Kingdom13 region13 For13 more13 informaon13 read13 how13 to13 provision13 a13 Bluemix13 virtual13 server13 including13 some13 simple13 setup13 steps13 for13 ease13 of13 use13
Page of 7 71 Page of 7 71
Create13 an13 SSH13 key13 To13 access13 your13 virtual13 server13 once13 it13 is13 provisioned13 allocate13 an13 SSH13 key13 to13 the13 virtual13 server13
To13 generate13 a13 publicprivate13 RSA13 key13 pair13 follow13 these13 steps13
1 In13 a13 UNIX13 or13 Linuxreg13 window13 run13 the13 following13 command13 specifying13 a13 ltuniquekeynamegt13 of13 your13 your13 choice13 and13 a13 password13 or13 passphrase13 (memorize13 the13 password)$13 ssh-shy‐keygen13 -shy‐t13 rsa13 -shy‐f13 ltuniquekeynamegt13
2 Enter13 the13 passphrase13 (empty13 for13 no13 passphrase)13
3 Enter13 the13 same13 passphrase13 again13
Your13 idenficaon13 has13 been13 saved13 in13 ltuniquekeynamegt13
Your13 public13 key13 has13 been13 saved13 in13 ltuniquekeynamegtpub13
The13 key13 fingerprint13 is13
d6268464506525930ce4632c693c203c13 ltusergtltservergt13
The13 keys13 randomart13 image13 is13
+--[ RSA 2048]----+
|o ++=o |
| E o ooo |
| |
| + o |
| S o |
| o |
| |
| |
| |
+-----------------+
Two13 keys13 are13 created13 public13 and13 private13 The13 public13 key13 contents13 are13 used13 to13 create13 a13 virtual13 server13 with13 a13 public13 IP13 address13 The13 private13 key13 is13 used13 to13 access13 the13 virtual13 server13 from13 a13 client13 machine13
Page of 8 71 Page of 8 71
Provisioning13 a13 virtual13 server13 1 A13 Bluemix13 account13 is13 required13 for13 this13 process13 Sign13 up13 here13 and13 log13 in13 Make13 sure13 your13
environment13 is13 correct13 For13 example13 if13 you13 are13 in13 the13 United13 Kingdom13 select13 the13 United13 Kingdom13 region13 in13 the13 Bluemix13 console13
13
13
2 Add13 a13 virtual13 server13 to13 your13 Bluemix13 environment13 Click13 Run13 Virtual13 Servers13
13
3 Select13 an13 OS13 image13 from13 the13 EXISTING13 list13 For13 ELK13 select13 Centos13 OS13 (use13 latest13 available13 version)13
Page of 9 71 Page of 9 71
4 Name13 the13 virtual13 server13
5 Make13 sure13 Assign13 public13 IP13 address13 is13 checked13
13
6 Click13 Add13 Key13 to13 add13 the13 SSH13 key13 you13 created13 previously13 Copy13 the13 contents13 of13 the13 public13 key13 into13 the13 Public13 Key13 to13 import13 field13
Page of 10 71 Page of 10 71
13
7 Click13 OK13 Then13 click13 CREATE13
The13 virtual13 server13 is13 provisioned13 and13 added13 to13 the13 Bluemix13 dashboard13 under13 VIRTUAL13 SERVERS13
Page of 11 71 Page of 11 71
13
Set13 up13 the13 virtual13 server13
Log13 on13 to13 the13 virtual13 server13 Now13 you13 can13 log13 on13 to13 the13 virtual13 server13 and13 add13 new13 users13 and13 programs13 to13 make13 it13 easier13 to13 access13 the13 virtual13 server13
To13 access13 the13 server13 go13 to13 the13 UNIX13 or13 Linux13 window13 that13 you13 used13 to13 create13 the13 SSH13 key13 In13 the13 directory13 where13 the13 ltuniquekeynamegt13 files13 exist13 run13 the13 following13 command13 referencing13 the13 SSH13 keyname13 the13 public13 IP13 address13 Bluemix13 allocated13 for13 the13 virtual13 server13 and13 the13 password13 for13 the13 SSH13 key13
[userserver ~]$ ssh -i ltuniquekeynamegt ibmcloudltpublicIPaddressgt
Enter13 the13 passphrase13 for13 the13 key13 ltuniquekeynamegt13
Last login Tue Jul 5 125546 2016 from 108-210-96-231lightspeedrlghncsbcglobalnet
[ibmcloudelkforcase ~]$
ibmcloud13 is13 the13 default13 user13 for13 the13 Bluemix13 virtual13 servers13 and13 allows13 SSH13 to13 access13 the13 server13
Acces13 the13 root13 user13 To13 access13 the13 root13 user13 log13 on13 as13 ibmcloud13 user13 and13 run13 the13 following13 command13
[ibmcloudelkforcase ~]$ sudo bash
[rootelkforcase ibmcloud]
The13 root13 user13 can13 be13 used13 to13 add13 programs13 to13 the13 server13 and13 create13 and13 edit13 users13 as13 required13 to13 access13 applicaons13 installed13 on13 the13 virtual13 server13
Page of 12 71 Page of 12 71
Copy13 files13 to13 the13 virtual13 server13 To13 move13 installaon13 files13 to13 the13 server13 go13 to13 the13 UNIX13 or13 Linux13 window13 you13 used13 to13 create13 the13 SSH13 key13 and13 the13 directory13 where13 the13 ltuniquekeynamegt13 files13 exist13 Move13 the13 installaon13 files13 to13 the13 same13 locaon13
Run13 the13 following13 command13 referencing13 the13 SSH13 keyname13 the13 public13 IP13 address13 Bluemix13 allocated13 for13 the13 virtual13 server13 and13 the13 password13 for13 the13 SSH13 key13
[userserver ~]$ scp ndashi ltuniquekeynamegt ltINSTALLATIONFILENAMEgt ibmcloudltpublicIPaddressgthomeibmcloud
Enter13 the13 passphrase13 for13 key13 cloudkey313
This13 copies13 the13 file13 using13 the13 ibmcloud13 user13 to13 the13 homeibmcloud13 directory13 on13 the13 virtual13 server13
Install13 and13 access13 the13 virtual13 server13 via13 VNC13 server13 Using13 the13 ssh13 command13 to13 access13 the13 virtual13 server13 is13 not13 the13 most13 stable13 method13 and13 does13 not13 offer13 X-shy‐Windows13 support13 Therefore13 we13 suggest13 that13 you13 use13 a13 VNC13 client13 to13 access13 the13 virtual13 server13
Installing13 a13 VNC13 client13 should13 be13 done13 as13 the13 root13 user13 Use13 the13 yum13 command13 as13 follows13
[ibmcloudelkforcase ~]$ yum groupinstall GNOME Desktop
[ibmcloudelkforcase ~]$ yum install tigervnc-server
These13 commands13 first13 install13 a13 desktop13 style13 OS13 user13 interface13 and13 then13 a13 VNC13 client13 to13 remotely13 access13 the13 desktop13 style13 OS13
Then13 as13 either13 the13 root13 user13 or13 another13 user13 (example13 below13 uses13 elk)13 run13 the13 following13 command13 to13 create13 the13 VNC13 client13 connecon13
[elkelkforcase ~]$ vncserver
New elkforcase2 (elk) desktop is elkforcase2
Starting applications specified in homeelkvncxstartup
Log file is homeelkvncelkforcase2log
[elkelkforcase ~]$
Install13 a13 VNC13 client13 on13 the13 client13 machine13 used13 to13 access13 the13 Internet13 and13 (in13 the13 example13 above)13 use13 the13 command13 ltpublicIPadddressgt213 and13 the13 password13 for13 ELK13 to13 access13 the13 virtual13 server13
Using13 a13 VNC13 client13 means13 that13 you13 can13 leave13 long13 duraon13 installaons13 running13 without13 fear13 of13 the13 terminal13 losing13 connecon13 and13 the13 installaons13 failing13
Install13 the13 ELK13 stack13 Download13 the13 source13 soaware13 code13 for13 the13 ELK13 stack13 from13 the13 ELK13 website13 13
At13 the13 me13 of13 wring13 the13 downloaded13 soaware13 files13 were13
Page of 13 71 Page of 13 71
13
Transfer13 the13 files13 to13 the13 virtual13 server13 (using13 the13 scp13 method13 detailed13 above)13
Install13 Elas0csearch13 As13 a13 root13 user13 create13 a13 non-shy‐root13 user13 with13 the13 adduser13 command13
Create13 a13 directory13 to13 install13 Elascsearch13 into13 and13 untar13 the13 contents13 of13 the13 Elascsearch13 install13 file13 into13 that13 directory13
[elkelkforcase]$ cd home
[elkelkforcase]$ mkdir elasticsearch
[elkelkforcase]$ cp homeibmcloudelasticsearch-233targz homeelasticsearch
[elkelkforcase]$ cd elasticsearch
[elkelkforcase]$ gunzip elasticsearch-233targz
[elkelkforcase]$ untar elasticsearch-233tar
Elascsearch13 is13 now13 installed13
Start13 Elas0csearch13 To13 start13 Elascsearch13 go13 to13 the13 bin13 directory13 as13 the13 non-shy‐root13 user13 when13 you13 installed13 it13 and13 run13 the13 elascsearch13 executable13
[elkelkforcase]$ cd homeelasticsearchelasticsearch-233bin
[elkelkforcase]$ elasticsearch amp
[2] 22211
[elkelkforcase]$ [2016-07-05 133857730][INFO ][node] [Arsenal] version[233] pid[22211] build[218bdf12016-05-17T154004Z]
[2016-07-05 133857740][INFO ][node] [Arsenal] initializing
[2016-07-05 133858984][INFO ][plugins] [Arsenal] modules [reindex lang-expression lang-groovy] plugins [] sites []
[2016-07-05 133859146][INFO ][env] [Arsenal] using [1] data paths mounts [[ (rootfs)]] net usable_space [155gb] net total_space [199gb] spins [unknown] types [rootfs]
[2016-07-05 133859146][INFO ][env] [Arsenal] heap size [10156mb] compressed ordinary object pointers [true]
Page of 14 71 Page of 14 71
[2016-07-05 133859147][WARN ][env] [Arsenal] max file descriptors [4096] for elasticsearch process likely too low consider increasing to at least [65536]
[2016-07-05 133903526][INFO ][node] [Arsenal] initialized
[2016-07-05 133903526][INFO ][node] [Arsenal] starting
To13 test13 that13 Elascsearch13 is13 running13 correctly13 open13 the13 browser13 in13 the13 VNC13 client13 and13 run13 localhost920013
13
Install13 Kibana13 ELK13 recommends13 installing13 and13 running13 Kibana13 as13 root13 13
1 As13 root13 user13 create13 an13 installaon13 directory13 and13 untar13 the13 installaon13 file13
[rootelkforcase]$ cd home
[rootelkforcase]$ mkdir kibana
[rootelkforcase]$ cp homeibmcloudkibana-451-linux-x64targz homekibana
[rootelkforcase]$ cd kibana
[rootelkforcase]$ gunzip kibana-451-linux-x64targz
[rootelkforcase]$ untar kibana-451-linux-x64tar
2 Modify13 the13 manifestyml13 configuraon13 file13 Go13 to13 the13 kibana-shy‐451-shy‐linux-shy‐x6413 directory13
[rootelkforcase]$ cd homekibanakibana-451-linux-x64
3 Edit13 the13 manifestyml13 file13 so13 the13 contents13 are13
applicaons13
-shy‐ name13 kibana-shy‐in-shy‐bluemix13
memory13 128M13
host13 kibana-shy‐in-shy‐bluemix13
buildpack13 hcpsgithubcomcloudfoundry-shy‐communitynginx-shy‐buildpackgit13
Page of 15 71 Page of 15 71
4 Edit13 the13 configkibanayml13 file13 so13 the13 contents13 are13
Kibana is served by a back end server This controls which port to use
port 5601
The host to bind the server to
host 0000
If you are running kibana behind a proxy and want to mount it at a path
specify that path here The basePath cant end in a slash
serverbasePath
The maximum payload size in bytes on incoming server requests
servermaxPayloadBytes 1048576
The Elasticsearch instance to use for all your queries
elasticsearchurl httplocalhost9200
preserve_elasticsearch_host true will send the hostname specified in `elasticsearch` If you set it to false
then the host you use to connect to this Kibana instance will be sent
elasticsearchpreserveHost true
Kibana uses an index in Elasticsearch to store saved searches visualizations
and dashboards It will create a new index if it doesnt already exist
kibanaindex kibana
The default application to load
kibanadefaultAppId discover
If your Elasticsearch is protected with basic auth these are the user credentials
used by the Kibana server to perform maintenance on the kibana_index at startup Your
Kibana users will still need to authenticate with Elasticsearch (which is proxied through
the Kibana server)
Page of 16 71 Page of 16 71
elasticsearchusername user
elasticsearchpassword pass
SSL for outgoing requests from the Kibana Server to the browser (PEM formatted)
serversslcert pathtoyourservercrt
serversslkey pathtoyourserverkey
Optional setting to validate that your Elasticsearch backend uses the same key files (PEM formatted)
elasticsearchsslcert pathtoyourclientcrt
elasticsearchsslkey pathtoyourclientkey
If you need to provide a CA certificate for your Elasticsearch instance put
the path of the pem file here
elasticsearchsslca pathtoyourCApem
Set to false to have a complete disregard for the validity of the SSL
certificate
elasticsearchsslverify true
Time in milliseconds to wait for elasticsearch to respond to pings defaults to
request_timeout setting
elasticsearchpingTimeout 1500
Time in milliseconds to wait for responses from the back end or elasticsearch
This must be gt 0
elasticsearchrequestTimeout 30000
Time in milliseconds for Elasticsearch to wait for responses from shards
Set to 0 to disable
elasticsearchshardTimeout 0
Time in milliseconds to wait for Elasticsearch at Kibana startup before retrying
elasticsearchstartupTimeout 5000
elasticsearchstartupTimeout 30000
Page of 17 71 Page of 17 71
Set the path to where you would like the process id file to be created
pidfile varrunkibanapid
If you would like to send the log output to a file you can set the path below
loggingdest stdout
Set this to true to suppress all logging output
loggingsilent false
Set this to true to suppress all logging output except for error messages
loggingquiet false
Set this to true to log all events including system usage information and all requests
loggingverbose false
Kibana13 is13 now13 installed13
Start13 Kibana13 To13 start13 Kibana13 go13 to13 the13 bin13 directory13 and13 run13 the13 kibana13 executable13
[rootelkforcase]$
[rootelkforcase]$ cd homekibanakibana-451-linux-x64bin
[rootelkforcase]$ kibana amp
[1] 22283
[rootelkforcase] log [095900542] [info][status][pluginkibana] Status changed from uninitialized to green - Ready
log [095900600] [info][status][pluginelasticsearch] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [095900689] [info][status][pluginkbn_vislib_vis_types] Status changed from uninitialized to green - Ready
log [095900717] [info][status][pluginmarkdown_vis] Status changed from uninitialized to green - Ready
log [095900721] [info][status][pluginmetric_vis] Status changed from uninitialized to green - Ready
log [095900814] [info][status][pluginspyModes] Status changed from uninitialized to green - Ready
log [095900934] [info][status][pluginstatusPage] Status changed from uninitialized to green - Ready
Page of 18 71 Page of 18 71
log [095900941] [info][status][plugintable_vis] Status changed from uninitialized to green - Ready
log [095901067] [info][listening] Server running at http00005601
log [095906083] [info][status][pluginelasticsearch] Status changed from yellow to yellow - No existing Kibana index found
log [095908942] [info][status][pluginelasticsearch] Status changed from yellow to green - Kibana index ready
Kibana13 is13 now13 started13
Broadcast13 Bluemix13 app13 log13 records13 Bluemix13 apps13 (which13 are13 shown13 in13 the13 US13 South13 Bluemix13 region13 of13 the13 System13 Architecture13 diagram)13 will13 produce13 logs13 in13 a13 syslog13 rfc542413 format13 using13 a13 Bluemix13 tool13 called13 Loggregator13
For13 Logstash13 to13 receive13 these13 logs13 each13 Bluemix13 app13 needs13 to13 broadcast13 these13 logs13 To13 do13 that13 run13 the13 following13 commands13 on13 each13 Bluemix13 app13 using13 the13 public13 IP13 address13 of13 the13 ELK13 virtual13 server13 the13 ELK13 tag13 to13 trace13 the13 process13 and13 insert13 the13 name13 of13 each13 app13 in13 turn13
cf cups elk -l syslogltpublicIPaddress5000 cf bind-service ltmyappgt elk
cf restage ltmyappgt
Install13 Logstash13 To13 install13 Logstash13 as13 root13 user13 create13 a13 directory13 for13 the13 installaon13 and13 untar13 the13 installaon13 file13 there13
13 13 [rootelkforcase]$ cd home
[rootelkforcase]$ mkdir logstash
[rootelkforcase]$ cp homeibmcloudlogstash-233targz homelogstash
[rootelkforcase]$ cd logstash
[rootelkforcase]$ gunzip logstash-233targz
[rootelkforcase]$ untar logstash-233tar
Write13 Logstash13 configura0on13 Logstash13 requires13 a13 configuraon13 file13 to13 be13 wricen13 to13 receive13 transform13 and13 output13 the13 log13 records13 to13 Elascsearch13 This13 secon13 explains13 the13 basic13 requirements13 of13 processing13 Bluemix13 app13 log13 records13 into13 Elascsearch13 (more13 complex13 configuraons13 are13 detailed13 later13 in13 this13 document13 for13 the13 integraons13 with13 third-shy‐party13 tools)13
Create13 a13 conf13 file13 as13 root13 user13
[rootelkforcase]$ cd homelogstashlogstash-233
[rootelkforcase]$ mkdir config
Page of 19 71 Page of 19 71
[rootelkforcase]$ cd config
[rootelkforcase]$ touch loggregatorconf
Edit13 the13 loggregator13 file13 with13 the13 following13 contents13
13 Input13 secon13 defines13 which13 log13 records13 to13 receive13
13 port13 and13 type13 reflect13 method13 selected13 for13 Cloud13 Foundry13 to13 broadcast13 the13 records13
input
tcp
port =gt 5000
type =gt syslog
13 Filter13 secon13 ndash13 transforms13 the13 data13
filter
Add a field to identify the records as Cloud Foundry based
mutate
add_field =gt format =gt cf
13 Output13 secon13 ndash13 sends13 the13 data13 to13 its13 desnaon(s)13
output
13 Use13 the13 elascsearch13 plugin13 to13 send13 the13 data13 to13 the13 elascsearch13 instance13
13 elasticsearch
hosts =gt localhost
stdout codec =gt rubydebug
13 End13 of13 Config13
Save13 the13 file13 Verify13 the13 format13 of13 the13 file13 is13 acceptable13 by13 running13 the13 following13 commands13
[rootelkforcase]$ cd homelogstashlogstash-233
[rootelkforcase]$ binlogstash agent --configtest -f configloggregatorconf
Page of 20 71 Page of 20 71
Configuraon13 is13 OK13
Start13 Logstash13 Aaer13 you13 have13 wricen13 and13 verified13 the13 Logstash13 configuraon13 you13 can13 start13 Logstash13 to13 process13 records13 from13 the13 Bluemix13 apps13 (via13 Cloud13 Foundry)13 Run13 the13 following13 as13 root13 user13
[rootelkforcase]$ cd homelogstashlogstash-233
[rootelkforcase]$ binlogstash agent -f configloggregatorconf
Logstash13 is13 now13 started13
Monitor13 Logstash13 processing13 Once13 Logstash13 is13 running13 and13 log13 records13 are13 being13 produced13 by13 the13 Bluemix13 apps13 you13 can13 monitor13 progress13 in13 the13 same13 window13 that13 was13 used13 to13 start13 Logstash13 Output13 is13 similar13 to13 the13 following13
13
The13 log13 records13 being13 broadcast13 from13 the13 Bluemix13 apps13 are13 now13 sent13 to13 Elascsearch13 from13 the13 Logstash13 output13 configuraon13 Records13 are13 sent13 in13 real13 me13 If13 Logstash13 is13 not13 running13 while13 log13 records13 are13 being13 produced13 then13 these13 records13 will13 not13 be13 processed13 by13 Logstash13 and13 will13 not13 be13 stored13 in13 Elascsearch13
Page of 21 71 Page of 21 71
Use13 ELK13 The13 log13 records13 from13 the13 Bluemix13 apps13 are13 now13 being13 stored13 and13 indexed13 by13 the13 Elascsearch13 component13 This13 data13 can13 be13 accessed13 via13 the13 Kibana13 applicaon13
Kibana13 Discover13 page13 Using13 a13 web13 browser13 either13 directly13 from13 a13 client13 or13 using13 the13 VNC13 client13 access13 Kibana13 using13 the13 publicIPaddress13 of13 the13 ELK13 virtual13 server13
hcpltpublicIPaddressgt560113
No13 userpassword13 credenals13 are13 required13
Kibana13 defaults13 to13 a13 search13 of13 all13 records13 ()13 in13 the13 last13 1513 minutes13 and13 returns13 the13 results13
13
More13 complex13 searches13 are13 possible13 by13 changing13 the13 me13 filter13 and13 supplying13 search13 queries13
Change13 the13 0me13 filter13 To13 change13 the13 me13 ranges13 click13 the13 clock13 symbol13 in13 the13 top-shy‐right13 of13 the13 Kibana13 Discover13 page13
13
You13 will13 see13 the13 following13 opons13
13
Page of 22 71 Page of 22 71
A13 variety13 of13 quick13 me13 ranges13 can13 be13 selected13 or13 you13 can13 switch13 to13 relave13 or13 absolute13 me13 ranges13
Quick13 me13 ranges13 allow13 switching13 between13 last13 X13 minutes13 to13 last13 Y13 hours13 or13 Z13 days13 etc13
Relave13 me13 ranges13 allow13 more13 specific13 versions13 of13 the13 Quick13 opons13 from13 specific13 minuteshoursdays13 to13 now13
13
Absolute13 me13 ranges13 specify13 the13 start13 and13 end13 mes13 for13 the13 searches13
13
Custom13 search13 queries13 You13 can13 use13 the13 text13 window13 of13 the13 Discover13 page13 to13 search13 for13 matches13 within13 the13 log13 records13 You13 can13 use13 Apache13 Lucene13 or13 Elascsearch13 query13 DSL13 syntax13
13
Page of 23 71 Page of 23 71
Saved13 searches13 You13 can13 save13 any13 custom13 search13 syntax13 and13 choice13 of13 me13 range13 for13 future13 ease13 of13 use13 Any13 relave13 or13 absolute13 me13 frame13 is13 saved13 as13 part13 of13 the13 search13 13
To13 save13 a13 search13
1 Set13 up13 and13 run13 the13 required13 syntax13 and13 me13 frame13
13
2 Click13 the13 Save13 Search13 icon13
13
3 Type13 in13 a13 name13 for13 the13 saved13 search13
Page of 24 71 Page of 24 71
13
7 To13 delete13 or13 update13 saved13 searches13 click13 13 manage13 saved13 searches13
13
Use13 the13 edit13 icons13 to13 make13 any13 changes13
Page of 26 71 Page of 26 71
13
Build13 simple13 Kibana13 histograms13 Kibana13 allows13 you13 to13 build13 charts13 and13 dashboards13 based13 on13 the13 search13 capability13 offered13 by13 the13 Discover13 page13 Saved13 searches13 can13 be13 used13 as13 the13 basis13 for13 charts13 and13 dashboards13 built13 using13 those13 charts13 To13 build13 a13 simple13 histogram13
1 Select13 the13 Visualize13 page13
13
2 Select13 Ver0cal13 Bar13 Chart13
13
3 To13 use13 a13 saved13 search13 click13 From13 a13 saved13 search13
Page of 27 71 Page of 27 71
13
4 Select13 a13 previously13 saved13 search13
13
5 Select13 to13 edit13 the13 x-shy‐axis
6 Edit13 the13 fields13 with13 informaon13 about13 the13 aggregaon13 field13 interval13 and13 custom13 label13
Page of 28 71 Page of 28 71
13
Page of 29 71 Page of 29 71
7 Click13 the13 green13 arrow13 The13 histogram13 is13 created13 and13 run13 based13 on13 the13 saved13 search
13
8 To13 save13 the13 histogram13 click13 the13 Save13 Visualiza0on13 icon13 and13 add13 a13 tle13
13
13
9 Click13 Save13
10 To13 view13 the13 visualisaon13 click13 the13 Dashboard13 page13
Page of 30 71 Page of 30 71
13
11 Click13 the13 Add13 Visualiza0on13 icon13
13
12 Select13 the13 visiualizaon13 that13 you13 just13 created13
13
Page of 31 71 Page of 31 71
13
13 To13 save13 the13 dashboard13 click13 the13 Save13 Dashboard13 icon13
Page of 32 71 Page of 32 71
13
14 Add13 a13 tle13 and13 select13 whether13 to13 store13 the13 selected13 me13 frame13 with13 the13 dashboard13
13
15 Click13 Save13
16 You13 can13 edit13 the13 shape13 and13 number13 of13 charts13 and13 save13 the13 dashboard13
17 To13 re-shy‐run13 the13 dashboard13 select13 the13 Dashboard13 page13 and13 click13 the13 Load13 Saved13 Dashboard13 icon13
Page of 33 71 Page of 33 71
13
18 Select13 the13 saved13 dashboard13 from13 the13 list13
13
19 The13 dashboard13 runs13
13
Page of 34 71 Page of 34 71
Page of 35 71 Page of 35 71
Aler0ng13 with13 ELK13 ELK13 examines13 the13 log13 records13 in13 process13 to13 send13 alerts13 to13 third-shy‐party13 alerng13 event13 management13 and13 social13 media13 applicaons13 As13 shown13 previously13 in13 the13 ELK13 architecture13 the13 ELK13 strategy13 for13 alerng13 is13 to13 send13 alerts13 from13 the13 Logstash13 component13 directly13 Therefore13 log13 records13 and13 alerts13 are13 sent13 to13 both13 Elascsearch13 and13 third-shy‐party13 tools13 at13 the13 same13 instant13 using13 the13 output13 secon13 of13 the13 Logstash13 configuraon13 Any13 alerts13 are13 generated13 during13 the13 filter13 secon13 processing13 of13 the13 Logstash13 configuraon13
Create13 alerts13 in13 Logstash13 The13 Logstash13 filter13 secon13 is13 used13 to13 trigger13 alerts13 based13 on13 the13 log13 records13 being13 processed13 For13 example13 log13 records13 can13 be13 checked13 for13 errors13 warnings13 crashed13 or13 stopped13 messages13 and13 generate13 an13 alert13
Here13 is13 an13 example13 Logstash13 filter13 code13
13 if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
The13 specific13 tags13 are13 set13 against13 the13 log13 records13 to13 trigger13 the13 alert13 later13 in13 the13 configuraon13 in13 the13 output13 secon13
Control13 the13 number13 of13 generated13 alerts13 using13 the13 Logstash13 throcle13 plugin13 so13 that13 either13 a13 lot13 of13 instances13 in13 a13 fixed13 me13 period13 are13 required13 before13 an13 alert13 is13 sent13 or13 a13 specific13 number13 of13 instances13 triggers13 the13 alert13
Page of 36 71 Page of 36 71
Edit13 the13 plugin13 parameters13 to13 meet13 system13 requirements13
13 13 13 13 13 if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 10
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle13
after_count =gt 20
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 30
period =gt 1200
key =gt servicename_alarm_warning
Page of 37 71 Page of 37 71
add_tag =gt servicename_throttled
13 13 13 13 13 13 13 13 13
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 40
period =gt 1500
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
Enrich13 alerts13 with13 message13 detail13 Some13 of13 the13 third-shy‐party13 integraons13 benefit13 from13 having13 components13 of13 the13 raw13 log13 record13 sent13 in13 addion13 to13 the13 raw13 record13 or13 fields13 in13 the13 record13 edited13 transformed13 or13 translated13
Grok13 Logstash13 allows13 the13 log13 record13 to13 be13 broken13 up13 into13 logical13 secons13 and13 each13 secon13 stored13 in13 a13 new13 parameter13 using13 the13 grok13 parameter13 in13 the13 filter13 secon13 Here13 is13 an13 example13 grok13
13 grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
To13 build13 the13 grok13 statement13 Logstash13 offers13 an13 online13 tool13 at13 hcpsgrokdebugherokuappcom13
Translate13 If13 specific13 fields13 in13 the13 log13 record13 are13 coded13 (for13 example13 the13 Bluemix13 app13 name13 is13 oaen13 not13 sent13 in13 the13 syslog13 messages13 instead13 the13 GUID13 is13 sent13 which13 is13 not13 useful)13 then13 the13 filter13 secon13 translate13 plugin13 can13 be13 used13 to13 add13 new13 fields13 by13 holding13 the13 decoded13 versions13 of13 those13 values13
13 translate
field =gt data1
destination =gt data1
override =gt true
Page of 38 71 Page of 38 71
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
dc50f6e4-c052-436a-bf27-95ba392c4228 monverifyservicemanagement
0c677e82-5bfc-4e38-9155-e416249e3ee6 ms-catalog-postpyretic-pomiculture
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
However13 the13 translate13 plugin13 is13 not13 included13 with13 the13 Logstash13 installaon13 In13 the13 next13 secon13 I13 will13 13 describe13 how13 to13 add13 new13 plugins13 to13 the13 Logstash13 instance13
Add13 custom13 plugins13 to13 Logstash13 As13 an13 open13 source13 tool13 Logstash13 encourages13 users13 to13 develop13 content13 for13 all13 users13 to13 share13 Some13 of13 the13 plugins13 that13 were13 created13 by13 our13 community13 include13 a13 translaon13 plugin13 an13 output13 Slack13 plugin13 and13 an13 output13 syslog13 plugin13 Letrsquos13 look13 at13 how13 to13 add13 these13 plugins13 to13 your13 environment13
Filter13 translate13 plugin13 To13 add13 the13 translate13 plugin13
1 Download13 the13 plugin13
2 Copy13 the13 plugin13 file13 logstash-shy‐filter-shy‐translate-shy‐214gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-filter-translate 214
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Output13 Slack13 plugin13 To13 add13 the13 Slack13 plugin13
Page of 39 71 Page of 39 71
1 Download13 the13 plugin13 13
2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐slack-shy‐014gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-output-slack 014
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Output13 syslog13 plugin13 To13 add13 the13 syslog13 plugin13
1 Download13 the13 plugin13 13
2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐syslog-shy‐214gem13 to13 the13 ELK13 Virtual13 Server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-output-syslog 214
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Integrate13 Logstash13 with13 Slack13 To13 send13 generated13 alerts13 to13 Slack13 complete13 these13 steps13
1 Create13 a13 Slack13 integraon13 13
2 Use13 the13 Slack13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 Slack13 integraon13
113 Login13 to13 Slack13 and13 go13 to13 the13 Apps13 page13 13
Page of 40 71 Page of 40 71
13
313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13
13
A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13
13
Page of 42 71 Page of 42 71
The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13
13
The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13
13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
slack
url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5
channel =gt cloud-operations-mon
username =gt nick_cawood
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13
Page of 43 71 Page of 43 71
Page of 44 71 Page of 44 71
In13 Slack13 the13 alerts13 look13 like13 this13
13
Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13
1 Create13 a13 PagerDuty13 integraon13 13
2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13
1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13
13
2 Click13 Add13 New13 Service13
13
313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13
Page of 45 71 Page of 45 71
13
413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13
513 Click13 Add13 Service13
13
The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13
Page of 46 71 Page of 46 71
13
The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt be7eb1ea9c224018923e818d3e891d0b
incident_key =gt logstashhosttype
Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13
Page of 47 71 Page of 47 71
13
Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13
1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13
2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13
3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13
4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13
13 13 http
url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo
http_method =gt post
format =gt json
For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13
13 13 syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
Page of 48 71 Page of 48 71
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13
13
The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13
if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2
This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13
Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13
Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13
Logstash13 severity Shown13 in13 Omnibus13 as
alert Indeterminate
crical Warning
debug 7
emergency Clear
error Minor
informaonal 6
noce Crical
warning Major
Page of 49 71 Page of 49 71
13
Page of 50 71 Page of 50 71
Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13
input
tcp
port =gt 5000
type =gt syslog
filter
mutate
add_field =gt format =gt cf
grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
translate
field =gt data1
destination =gt data1
override =gt true
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
=== Check Alarm Criteria ===
Standard Key Words
if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
=== Alarm Exclusions ===
if [message] =~ (i)sometexthere
mutate remove_tag =gt servicename_alarm
Throttle alarms so we do not hit PD rate limits
Page of 52 71 Page of 52 71
if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 5
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle
after_count =gt 5
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 5
period =gt 1200
key =gt servicename_alarm_warning
add_tag =gt servicename_throttled
Page of 53 71 Page of 53 71
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
output
elasticsearch
hosts =gt localhost
stdout codec =gt rubydebug
if servicename_alarm_err in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
Page of 54 71 Page of 54 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 55 71 Page of 55 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
Page of 56 71 Page of 56 71
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert err message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_error in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
Page of 57 71 Page of 57 71
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 58 71 Page of 58 71
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 59 71 Page of 59 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert error message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
Page of 60 71 Page of 60 71
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
Page of 61 71 Page of 61 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 62 71 Page of 62 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt x
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert fatal message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
Page of 63 71 Page of 63 71
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
Page of 64 71 Page of 64 71
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 65 71 Page of 65 71
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt heavy_exclamation_mark
Page of 66 71 Page of 66 71
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert WARNING message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 67 71 Page of 67 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
Page of 68 71 Page of 68 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 69 71 Page of 69 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
Page of 70 71 Page of 70 71
Page of 71 71 Page of 71 71
Service13 Management13 architecture13 The13 IBM13 Cloud13 Architecture13 Center13 created13 a13 standard13 reference13 architecture13 for13 how13 to13 set13 up13 your13 cloud13 environemnt13 to13 handle13 incident13 management13 13
View13 the13 standard13 Service13 Management13 architecure13
13 13
Log13 Monitoring13 System13 architecture13 The13 following13 diagram13 shows13 a13 standard13 architecture13 for13 the13 log-shy‐monitoring13 system13 described13 in13 this13 tutorial13 13
13
Bluemix13 environment13 system13 architecture13 The13 ELK13 stack13 can13 be13 installed13 into13 a13 variety13 of13 environments13 When13 using13 Bluemix13 the13 following13 figure13 shows13 what13 the13 environment13 might13 look13 like13
Page of 6 71 Page of 6 71
13
The13 Bluemix13 cloud13 plaRorm13 offers13 a13 Public13 network13 and13 access13 to13 US13 South13 and13 UK13 environments13 From13 all13 these13 environments13 third-shy‐party13 cloud-shy‐based13 soluons13 are13 also13 accessible13
The13 ELK13 stack13 is13 installed13 in13 the13 Bluemix13 United13 Kingdom13 environment13 because13 this13 is13 the13 environment13 where13 Bluemix13 has13 virtual13 servers13 available13 for13 provisioning13 Other13 applicaons13 in13 the13 CASE13 project13 were13 also13 installed13 on13 virtual13 servers13 in13 the13 Bluemix13 United13 Kingdom13 region13
Bluemix13 apps13 were13 used13 to13 create13 a13 system13 to13 be13 monitored13 by13 ELK13 These13 apps13 were13 hosted13 in13 the13 Bluemix13 US13 South13 environment13 using13 Bluemix13 microservices13 which13 produce13 logs13 during13 normal13 operaon13 and13 during13 crashes13 and13 outages13
Through13 the13 Bluemix13 Cloud13 Foundry13 process13 the13 log13 records13 can13 be13 broadcast13 from13 the13 Bluemix13 apps13 so13 Logstash13 can13 receive13 them13 and13 send13 them13 to13 Elascsearch13 within13 the13 ELK13 stack13 or13 other13 virtual13 servers13 (hosng13 other13 IBM13 or13 third-shy‐party13 products)13 in13 the13 Bluemix13 United13 Kingdom13 environment13 or13 to13 third-shy‐party13 applicaons13 in13 the13 public13 cloud13 (Pager13 Duty13 Slack13 etc)13
Crea0ng13 a13 virtual13 server13 in13 Bluemix13 With13 Bluemix13 you13 can13 provision13 a13 virtual13 server13 in13 the13 cloud13 to13 install13 standalone13 or13 integrated13 system13 components13 Not13 all13 Bluemix13 regions13 offer13 this13 service13 all13 the13 me13 due13 to13 beta-shy‐project13 constraints13 In13 this13 arclersquos13 examples13 virtual13 servers13 were13 allowed13 to13 be13 provisioned13 in13 the13 United13 Kingdom13 region13 For13 more13 informaon13 read13 how13 to13 provision13 a13 Bluemix13 virtual13 server13 including13 some13 simple13 setup13 steps13 for13 ease13 of13 use13
Page of 7 71 Page of 7 71
Create13 an13 SSH13 key13 To13 access13 your13 virtual13 server13 once13 it13 is13 provisioned13 allocate13 an13 SSH13 key13 to13 the13 virtual13 server13
To13 generate13 a13 publicprivate13 RSA13 key13 pair13 follow13 these13 steps13
1 In13 a13 UNIX13 or13 Linuxreg13 window13 run13 the13 following13 command13 specifying13 a13 ltuniquekeynamegt13 of13 your13 your13 choice13 and13 a13 password13 or13 passphrase13 (memorize13 the13 password)$13 ssh-shy‐keygen13 -shy‐t13 rsa13 -shy‐f13 ltuniquekeynamegt13
2 Enter13 the13 passphrase13 (empty13 for13 no13 passphrase)13
3 Enter13 the13 same13 passphrase13 again13
Your13 idenficaon13 has13 been13 saved13 in13 ltuniquekeynamegt13
Your13 public13 key13 has13 been13 saved13 in13 ltuniquekeynamegtpub13
The13 key13 fingerprint13 is13
d6268464506525930ce4632c693c203c13 ltusergtltservergt13
The13 keys13 randomart13 image13 is13
+--[ RSA 2048]----+
|o ++=o |
| E o ooo |
| |
| + o |
| S o |
| o |
| |
| |
| |
+-----------------+
Two13 keys13 are13 created13 public13 and13 private13 The13 public13 key13 contents13 are13 used13 to13 create13 a13 virtual13 server13 with13 a13 public13 IP13 address13 The13 private13 key13 is13 used13 to13 access13 the13 virtual13 server13 from13 a13 client13 machine13
Page of 8 71 Page of 8 71
Provisioning13 a13 virtual13 server13 1 A13 Bluemix13 account13 is13 required13 for13 this13 process13 Sign13 up13 here13 and13 log13 in13 Make13 sure13 your13
environment13 is13 correct13 For13 example13 if13 you13 are13 in13 the13 United13 Kingdom13 select13 the13 United13 Kingdom13 region13 in13 the13 Bluemix13 console13
13
13
2 Add13 a13 virtual13 server13 to13 your13 Bluemix13 environment13 Click13 Run13 Virtual13 Servers13
13
3 Select13 an13 OS13 image13 from13 the13 EXISTING13 list13 For13 ELK13 select13 Centos13 OS13 (use13 latest13 available13 version)13
Page of 9 71 Page of 9 71
4 Name13 the13 virtual13 server13
5 Make13 sure13 Assign13 public13 IP13 address13 is13 checked13
13
6 Click13 Add13 Key13 to13 add13 the13 SSH13 key13 you13 created13 previously13 Copy13 the13 contents13 of13 the13 public13 key13 into13 the13 Public13 Key13 to13 import13 field13
Page of 10 71 Page of 10 71
13
7 Click13 OK13 Then13 click13 CREATE13
The13 virtual13 server13 is13 provisioned13 and13 added13 to13 the13 Bluemix13 dashboard13 under13 VIRTUAL13 SERVERS13
Page of 11 71 Page of 11 71
13
Set13 up13 the13 virtual13 server13
Log13 on13 to13 the13 virtual13 server13 Now13 you13 can13 log13 on13 to13 the13 virtual13 server13 and13 add13 new13 users13 and13 programs13 to13 make13 it13 easier13 to13 access13 the13 virtual13 server13
To13 access13 the13 server13 go13 to13 the13 UNIX13 or13 Linux13 window13 that13 you13 used13 to13 create13 the13 SSH13 key13 In13 the13 directory13 where13 the13 ltuniquekeynamegt13 files13 exist13 run13 the13 following13 command13 referencing13 the13 SSH13 keyname13 the13 public13 IP13 address13 Bluemix13 allocated13 for13 the13 virtual13 server13 and13 the13 password13 for13 the13 SSH13 key13
[userserver ~]$ ssh -i ltuniquekeynamegt ibmcloudltpublicIPaddressgt
Enter13 the13 passphrase13 for13 the13 key13 ltuniquekeynamegt13
Last login Tue Jul 5 125546 2016 from 108-210-96-231lightspeedrlghncsbcglobalnet
[ibmcloudelkforcase ~]$
ibmcloud13 is13 the13 default13 user13 for13 the13 Bluemix13 virtual13 servers13 and13 allows13 SSH13 to13 access13 the13 server13
Acces13 the13 root13 user13 To13 access13 the13 root13 user13 log13 on13 as13 ibmcloud13 user13 and13 run13 the13 following13 command13
[ibmcloudelkforcase ~]$ sudo bash
[rootelkforcase ibmcloud]
The13 root13 user13 can13 be13 used13 to13 add13 programs13 to13 the13 server13 and13 create13 and13 edit13 users13 as13 required13 to13 access13 applicaons13 installed13 on13 the13 virtual13 server13
Page of 12 71 Page of 12 71
Copy13 files13 to13 the13 virtual13 server13 To13 move13 installaon13 files13 to13 the13 server13 go13 to13 the13 UNIX13 or13 Linux13 window13 you13 used13 to13 create13 the13 SSH13 key13 and13 the13 directory13 where13 the13 ltuniquekeynamegt13 files13 exist13 Move13 the13 installaon13 files13 to13 the13 same13 locaon13
Run13 the13 following13 command13 referencing13 the13 SSH13 keyname13 the13 public13 IP13 address13 Bluemix13 allocated13 for13 the13 virtual13 server13 and13 the13 password13 for13 the13 SSH13 key13
[userserver ~]$ scp ndashi ltuniquekeynamegt ltINSTALLATIONFILENAMEgt ibmcloudltpublicIPaddressgthomeibmcloud
Enter13 the13 passphrase13 for13 key13 cloudkey313
This13 copies13 the13 file13 using13 the13 ibmcloud13 user13 to13 the13 homeibmcloud13 directory13 on13 the13 virtual13 server13
Install13 and13 access13 the13 virtual13 server13 via13 VNC13 server13 Using13 the13 ssh13 command13 to13 access13 the13 virtual13 server13 is13 not13 the13 most13 stable13 method13 and13 does13 not13 offer13 X-shy‐Windows13 support13 Therefore13 we13 suggest13 that13 you13 use13 a13 VNC13 client13 to13 access13 the13 virtual13 server13
Installing13 a13 VNC13 client13 should13 be13 done13 as13 the13 root13 user13 Use13 the13 yum13 command13 as13 follows13
[ibmcloudelkforcase ~]$ yum groupinstall GNOME Desktop
[ibmcloudelkforcase ~]$ yum install tigervnc-server
These13 commands13 first13 install13 a13 desktop13 style13 OS13 user13 interface13 and13 then13 a13 VNC13 client13 to13 remotely13 access13 the13 desktop13 style13 OS13
Then13 as13 either13 the13 root13 user13 or13 another13 user13 (example13 below13 uses13 elk)13 run13 the13 following13 command13 to13 create13 the13 VNC13 client13 connecon13
[elkelkforcase ~]$ vncserver
New elkforcase2 (elk) desktop is elkforcase2
Starting applications specified in homeelkvncxstartup
Log file is homeelkvncelkforcase2log
[elkelkforcase ~]$
Install13 a13 VNC13 client13 on13 the13 client13 machine13 used13 to13 access13 the13 Internet13 and13 (in13 the13 example13 above)13 use13 the13 command13 ltpublicIPadddressgt213 and13 the13 password13 for13 ELK13 to13 access13 the13 virtual13 server13
Using13 a13 VNC13 client13 means13 that13 you13 can13 leave13 long13 duraon13 installaons13 running13 without13 fear13 of13 the13 terminal13 losing13 connecon13 and13 the13 installaons13 failing13
Install13 the13 ELK13 stack13 Download13 the13 source13 soaware13 code13 for13 the13 ELK13 stack13 from13 the13 ELK13 website13 13
At13 the13 me13 of13 wring13 the13 downloaded13 soaware13 files13 were13
Page of 13 71 Page of 13 71
13
Transfer13 the13 files13 to13 the13 virtual13 server13 (using13 the13 scp13 method13 detailed13 above)13
Install13 Elas0csearch13 As13 a13 root13 user13 create13 a13 non-shy‐root13 user13 with13 the13 adduser13 command13
Create13 a13 directory13 to13 install13 Elascsearch13 into13 and13 untar13 the13 contents13 of13 the13 Elascsearch13 install13 file13 into13 that13 directory13
[elkelkforcase]$ cd home
[elkelkforcase]$ mkdir elasticsearch
[elkelkforcase]$ cp homeibmcloudelasticsearch-233targz homeelasticsearch
[elkelkforcase]$ cd elasticsearch
[elkelkforcase]$ gunzip elasticsearch-233targz
[elkelkforcase]$ untar elasticsearch-233tar
Elascsearch13 is13 now13 installed13
Start13 Elas0csearch13 To13 start13 Elascsearch13 go13 to13 the13 bin13 directory13 as13 the13 non-shy‐root13 user13 when13 you13 installed13 it13 and13 run13 the13 elascsearch13 executable13
[elkelkforcase]$ cd homeelasticsearchelasticsearch-233bin
[elkelkforcase]$ elasticsearch amp
[2] 22211
[elkelkforcase]$ [2016-07-05 133857730][INFO ][node] [Arsenal] version[233] pid[22211] build[218bdf12016-05-17T154004Z]
[2016-07-05 133857740][INFO ][node] [Arsenal] initializing
[2016-07-05 133858984][INFO ][plugins] [Arsenal] modules [reindex lang-expression lang-groovy] plugins [] sites []
[2016-07-05 133859146][INFO ][env] [Arsenal] using [1] data paths mounts [[ (rootfs)]] net usable_space [155gb] net total_space [199gb] spins [unknown] types [rootfs]
[2016-07-05 133859146][INFO ][env] [Arsenal] heap size [10156mb] compressed ordinary object pointers [true]
Page of 14 71 Page of 14 71
[2016-07-05 133859147][WARN ][env] [Arsenal] max file descriptors [4096] for elasticsearch process likely too low consider increasing to at least [65536]
[2016-07-05 133903526][INFO ][node] [Arsenal] initialized
[2016-07-05 133903526][INFO ][node] [Arsenal] starting
To13 test13 that13 Elascsearch13 is13 running13 correctly13 open13 the13 browser13 in13 the13 VNC13 client13 and13 run13 localhost920013
13
Install13 Kibana13 ELK13 recommends13 installing13 and13 running13 Kibana13 as13 root13 13
1 As13 root13 user13 create13 an13 installaon13 directory13 and13 untar13 the13 installaon13 file13
[rootelkforcase]$ cd home
[rootelkforcase]$ mkdir kibana
[rootelkforcase]$ cp homeibmcloudkibana-451-linux-x64targz homekibana
[rootelkforcase]$ cd kibana
[rootelkforcase]$ gunzip kibana-451-linux-x64targz
[rootelkforcase]$ untar kibana-451-linux-x64tar
2 Modify13 the13 manifestyml13 configuraon13 file13 Go13 to13 the13 kibana-shy‐451-shy‐linux-shy‐x6413 directory13
[rootelkforcase]$ cd homekibanakibana-451-linux-x64
3 Edit13 the13 manifestyml13 file13 so13 the13 contents13 are13
applicaons13
-shy‐ name13 kibana-shy‐in-shy‐bluemix13
memory13 128M13
host13 kibana-shy‐in-shy‐bluemix13
buildpack13 hcpsgithubcomcloudfoundry-shy‐communitynginx-shy‐buildpackgit13
Page of 15 71 Page of 15 71
4 Edit13 the13 configkibanayml13 file13 so13 the13 contents13 are13
Kibana is served by a back end server This controls which port to use
port 5601
The host to bind the server to
host 0000
If you are running kibana behind a proxy and want to mount it at a path
specify that path here The basePath cant end in a slash
serverbasePath
The maximum payload size in bytes on incoming server requests
servermaxPayloadBytes 1048576
The Elasticsearch instance to use for all your queries
elasticsearchurl httplocalhost9200
preserve_elasticsearch_host true will send the hostname specified in `elasticsearch` If you set it to false
then the host you use to connect to this Kibana instance will be sent
elasticsearchpreserveHost true
Kibana uses an index in Elasticsearch to store saved searches visualizations
and dashboards It will create a new index if it doesnt already exist
kibanaindex kibana
The default application to load
kibanadefaultAppId discover
If your Elasticsearch is protected with basic auth these are the user credentials
used by the Kibana server to perform maintenance on the kibana_index at startup Your
Kibana users will still need to authenticate with Elasticsearch (which is proxied through
the Kibana server)
Page of 16 71 Page of 16 71
elasticsearchusername user
elasticsearchpassword pass
SSL for outgoing requests from the Kibana Server to the browser (PEM formatted)
serversslcert pathtoyourservercrt
serversslkey pathtoyourserverkey
Optional setting to validate that your Elasticsearch backend uses the same key files (PEM formatted)
elasticsearchsslcert pathtoyourclientcrt
elasticsearchsslkey pathtoyourclientkey
If you need to provide a CA certificate for your Elasticsearch instance put
the path of the pem file here
elasticsearchsslca pathtoyourCApem
Set to false to have a complete disregard for the validity of the SSL
certificate
elasticsearchsslverify true
Time in milliseconds to wait for elasticsearch to respond to pings defaults to
request_timeout setting
elasticsearchpingTimeout 1500
Time in milliseconds to wait for responses from the back end or elasticsearch
This must be gt 0
elasticsearchrequestTimeout 30000
Time in milliseconds for Elasticsearch to wait for responses from shards
Set to 0 to disable
elasticsearchshardTimeout 0
Time in milliseconds to wait for Elasticsearch at Kibana startup before retrying
elasticsearchstartupTimeout 5000
elasticsearchstartupTimeout 30000
Page of 17 71 Page of 17 71
Set the path to where you would like the process id file to be created
pidfile varrunkibanapid
If you would like to send the log output to a file you can set the path below
loggingdest stdout
Set this to true to suppress all logging output
loggingsilent false
Set this to true to suppress all logging output except for error messages
loggingquiet false
Set this to true to log all events including system usage information and all requests
loggingverbose false
Kibana13 is13 now13 installed13
Start13 Kibana13 To13 start13 Kibana13 go13 to13 the13 bin13 directory13 and13 run13 the13 kibana13 executable13
[rootelkforcase]$
[rootelkforcase]$ cd homekibanakibana-451-linux-x64bin
[rootelkforcase]$ kibana amp
[1] 22283
[rootelkforcase] log [095900542] [info][status][pluginkibana] Status changed from uninitialized to green - Ready
log [095900600] [info][status][pluginelasticsearch] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [095900689] [info][status][pluginkbn_vislib_vis_types] Status changed from uninitialized to green - Ready
log [095900717] [info][status][pluginmarkdown_vis] Status changed from uninitialized to green - Ready
log [095900721] [info][status][pluginmetric_vis] Status changed from uninitialized to green - Ready
log [095900814] [info][status][pluginspyModes] Status changed from uninitialized to green - Ready
log [095900934] [info][status][pluginstatusPage] Status changed from uninitialized to green - Ready
Page of 18 71 Page of 18 71
log [095900941] [info][status][plugintable_vis] Status changed from uninitialized to green - Ready
log [095901067] [info][listening] Server running at http00005601
log [095906083] [info][status][pluginelasticsearch] Status changed from yellow to yellow - No existing Kibana index found
log [095908942] [info][status][pluginelasticsearch] Status changed from yellow to green - Kibana index ready
Kibana13 is13 now13 started13
Broadcast13 Bluemix13 app13 log13 records13 Bluemix13 apps13 (which13 are13 shown13 in13 the13 US13 South13 Bluemix13 region13 of13 the13 System13 Architecture13 diagram)13 will13 produce13 logs13 in13 a13 syslog13 rfc542413 format13 using13 a13 Bluemix13 tool13 called13 Loggregator13
For13 Logstash13 to13 receive13 these13 logs13 each13 Bluemix13 app13 needs13 to13 broadcast13 these13 logs13 To13 do13 that13 run13 the13 following13 commands13 on13 each13 Bluemix13 app13 using13 the13 public13 IP13 address13 of13 the13 ELK13 virtual13 server13 the13 ELK13 tag13 to13 trace13 the13 process13 and13 insert13 the13 name13 of13 each13 app13 in13 turn13
cf cups elk -l syslogltpublicIPaddress5000 cf bind-service ltmyappgt elk
cf restage ltmyappgt
Install13 Logstash13 To13 install13 Logstash13 as13 root13 user13 create13 a13 directory13 for13 the13 installaon13 and13 untar13 the13 installaon13 file13 there13
13 13 [rootelkforcase]$ cd home
[rootelkforcase]$ mkdir logstash
[rootelkforcase]$ cp homeibmcloudlogstash-233targz homelogstash
[rootelkforcase]$ cd logstash
[rootelkforcase]$ gunzip logstash-233targz
[rootelkforcase]$ untar logstash-233tar
Write13 Logstash13 configura0on13 Logstash13 requires13 a13 configuraon13 file13 to13 be13 wricen13 to13 receive13 transform13 and13 output13 the13 log13 records13 to13 Elascsearch13 This13 secon13 explains13 the13 basic13 requirements13 of13 processing13 Bluemix13 app13 log13 records13 into13 Elascsearch13 (more13 complex13 configuraons13 are13 detailed13 later13 in13 this13 document13 for13 the13 integraons13 with13 third-shy‐party13 tools)13
Create13 a13 conf13 file13 as13 root13 user13
[rootelkforcase]$ cd homelogstashlogstash-233
[rootelkforcase]$ mkdir config
Page of 19 71 Page of 19 71
[rootelkforcase]$ cd config
[rootelkforcase]$ touch loggregatorconf
Edit13 the13 loggregator13 file13 with13 the13 following13 contents13
13 Input13 secon13 defines13 which13 log13 records13 to13 receive13
13 port13 and13 type13 reflect13 method13 selected13 for13 Cloud13 Foundry13 to13 broadcast13 the13 records13
input
tcp
port =gt 5000
type =gt syslog
13 Filter13 secon13 ndash13 transforms13 the13 data13
filter
Add a field to identify the records as Cloud Foundry based
mutate
add_field =gt format =gt cf
13 Output13 secon13 ndash13 sends13 the13 data13 to13 its13 desnaon(s)13
output
13 Use13 the13 elascsearch13 plugin13 to13 send13 the13 data13 to13 the13 elascsearch13 instance13
13 elasticsearch
hosts =gt localhost
stdout codec =gt rubydebug
13 End13 of13 Config13
Save13 the13 file13 Verify13 the13 format13 of13 the13 file13 is13 acceptable13 by13 running13 the13 following13 commands13
[rootelkforcase]$ cd homelogstashlogstash-233
[rootelkforcase]$ binlogstash agent --configtest -f configloggregatorconf
Page of 20 71 Page of 20 71
Configuraon13 is13 OK13
Start13 Logstash13 Aaer13 you13 have13 wricen13 and13 verified13 the13 Logstash13 configuraon13 you13 can13 start13 Logstash13 to13 process13 records13 from13 the13 Bluemix13 apps13 (via13 Cloud13 Foundry)13 Run13 the13 following13 as13 root13 user13
[rootelkforcase]$ cd homelogstashlogstash-233
[rootelkforcase]$ binlogstash agent -f configloggregatorconf
Logstash13 is13 now13 started13
Monitor13 Logstash13 processing13 Once13 Logstash13 is13 running13 and13 log13 records13 are13 being13 produced13 by13 the13 Bluemix13 apps13 you13 can13 monitor13 progress13 in13 the13 same13 window13 that13 was13 used13 to13 start13 Logstash13 Output13 is13 similar13 to13 the13 following13
13
The13 log13 records13 being13 broadcast13 from13 the13 Bluemix13 apps13 are13 now13 sent13 to13 Elascsearch13 from13 the13 Logstash13 output13 configuraon13 Records13 are13 sent13 in13 real13 me13 If13 Logstash13 is13 not13 running13 while13 log13 records13 are13 being13 produced13 then13 these13 records13 will13 not13 be13 processed13 by13 Logstash13 and13 will13 not13 be13 stored13 in13 Elascsearch13
Page of 21 71 Page of 21 71
Use13 ELK13 The13 log13 records13 from13 the13 Bluemix13 apps13 are13 now13 being13 stored13 and13 indexed13 by13 the13 Elascsearch13 component13 This13 data13 can13 be13 accessed13 via13 the13 Kibana13 applicaon13
Kibana13 Discover13 page13 Using13 a13 web13 browser13 either13 directly13 from13 a13 client13 or13 using13 the13 VNC13 client13 access13 Kibana13 using13 the13 publicIPaddress13 of13 the13 ELK13 virtual13 server13
hcpltpublicIPaddressgt560113
No13 userpassword13 credenals13 are13 required13
Kibana13 defaults13 to13 a13 search13 of13 all13 records13 ()13 in13 the13 last13 1513 minutes13 and13 returns13 the13 results13
13
More13 complex13 searches13 are13 possible13 by13 changing13 the13 me13 filter13 and13 supplying13 search13 queries13
Change13 the13 0me13 filter13 To13 change13 the13 me13 ranges13 click13 the13 clock13 symbol13 in13 the13 top-shy‐right13 of13 the13 Kibana13 Discover13 page13
13
You13 will13 see13 the13 following13 opons13
13
Page of 22 71 Page of 22 71
A13 variety13 of13 quick13 me13 ranges13 can13 be13 selected13 or13 you13 can13 switch13 to13 relave13 or13 absolute13 me13 ranges13
Quick13 me13 ranges13 allow13 switching13 between13 last13 X13 minutes13 to13 last13 Y13 hours13 or13 Z13 days13 etc13
Relave13 me13 ranges13 allow13 more13 specific13 versions13 of13 the13 Quick13 opons13 from13 specific13 minuteshoursdays13 to13 now13
13
Absolute13 me13 ranges13 specify13 the13 start13 and13 end13 mes13 for13 the13 searches13
13
Custom13 search13 queries13 You13 can13 use13 the13 text13 window13 of13 the13 Discover13 page13 to13 search13 for13 matches13 within13 the13 log13 records13 You13 can13 use13 Apache13 Lucene13 or13 Elascsearch13 query13 DSL13 syntax13
13
Page of 23 71 Page of 23 71
Saved13 searches13 You13 can13 save13 any13 custom13 search13 syntax13 and13 choice13 of13 me13 range13 for13 future13 ease13 of13 use13 Any13 relave13 or13 absolute13 me13 frame13 is13 saved13 as13 part13 of13 the13 search13 13
To13 save13 a13 search13
1 Set13 up13 and13 run13 the13 required13 syntax13 and13 me13 frame13
13
2 Click13 the13 Save13 Search13 icon13
13
3 Type13 in13 a13 name13 for13 the13 saved13 search13
Page of 24 71 Page of 24 71
13
7 To13 delete13 or13 update13 saved13 searches13 click13 13 manage13 saved13 searches13
13
Use13 the13 edit13 icons13 to13 make13 any13 changes13
Page of 26 71 Page of 26 71
13
Build13 simple13 Kibana13 histograms13 Kibana13 allows13 you13 to13 build13 charts13 and13 dashboards13 based13 on13 the13 search13 capability13 offered13 by13 the13 Discover13 page13 Saved13 searches13 can13 be13 used13 as13 the13 basis13 for13 charts13 and13 dashboards13 built13 using13 those13 charts13 To13 build13 a13 simple13 histogram13
1 Select13 the13 Visualize13 page13
13
2 Select13 Ver0cal13 Bar13 Chart13
13
3 To13 use13 a13 saved13 search13 click13 From13 a13 saved13 search13
Page of 27 71 Page of 27 71
13
4 Select13 a13 previously13 saved13 search13
13
5 Select13 to13 edit13 the13 x-shy‐axis
6 Edit13 the13 fields13 with13 informaon13 about13 the13 aggregaon13 field13 interval13 and13 custom13 label13
Page of 28 71 Page of 28 71
13
Page of 29 71 Page of 29 71
7 Click13 the13 green13 arrow13 The13 histogram13 is13 created13 and13 run13 based13 on13 the13 saved13 search
13
8 To13 save13 the13 histogram13 click13 the13 Save13 Visualiza0on13 icon13 and13 add13 a13 tle13
13
13
9 Click13 Save13
10 To13 view13 the13 visualisaon13 click13 the13 Dashboard13 page13
Page of 30 71 Page of 30 71
13
11 Click13 the13 Add13 Visualiza0on13 icon13
13
12 Select13 the13 visiualizaon13 that13 you13 just13 created13
13
Page of 31 71 Page of 31 71
13
13 To13 save13 the13 dashboard13 click13 the13 Save13 Dashboard13 icon13
Page of 32 71 Page of 32 71
13
14 Add13 a13 tle13 and13 select13 whether13 to13 store13 the13 selected13 me13 frame13 with13 the13 dashboard13
13
15 Click13 Save13
16 You13 can13 edit13 the13 shape13 and13 number13 of13 charts13 and13 save13 the13 dashboard13
17 To13 re-shy‐run13 the13 dashboard13 select13 the13 Dashboard13 page13 and13 click13 the13 Load13 Saved13 Dashboard13 icon13
Page of 33 71 Page of 33 71
13
18 Select13 the13 saved13 dashboard13 from13 the13 list13
13
19 The13 dashboard13 runs13
13
Page of 34 71 Page of 34 71
Page of 35 71 Page of 35 71
Aler0ng13 with13 ELK13 ELK13 examines13 the13 log13 records13 in13 process13 to13 send13 alerts13 to13 third-shy‐party13 alerng13 event13 management13 and13 social13 media13 applicaons13 As13 shown13 previously13 in13 the13 ELK13 architecture13 the13 ELK13 strategy13 for13 alerng13 is13 to13 send13 alerts13 from13 the13 Logstash13 component13 directly13 Therefore13 log13 records13 and13 alerts13 are13 sent13 to13 both13 Elascsearch13 and13 third-shy‐party13 tools13 at13 the13 same13 instant13 using13 the13 output13 secon13 of13 the13 Logstash13 configuraon13 Any13 alerts13 are13 generated13 during13 the13 filter13 secon13 processing13 of13 the13 Logstash13 configuraon13
Create13 alerts13 in13 Logstash13 The13 Logstash13 filter13 secon13 is13 used13 to13 trigger13 alerts13 based13 on13 the13 log13 records13 being13 processed13 For13 example13 log13 records13 can13 be13 checked13 for13 errors13 warnings13 crashed13 or13 stopped13 messages13 and13 generate13 an13 alert13
Here13 is13 an13 example13 Logstash13 filter13 code13
13 if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
The13 specific13 tags13 are13 set13 against13 the13 log13 records13 to13 trigger13 the13 alert13 later13 in13 the13 configuraon13 in13 the13 output13 secon13
Control13 the13 number13 of13 generated13 alerts13 using13 the13 Logstash13 throcle13 plugin13 so13 that13 either13 a13 lot13 of13 instances13 in13 a13 fixed13 me13 period13 are13 required13 before13 an13 alert13 is13 sent13 or13 a13 specific13 number13 of13 instances13 triggers13 the13 alert13
Page of 36 71 Page of 36 71
Edit13 the13 plugin13 parameters13 to13 meet13 system13 requirements13
13 13 13 13 13 if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 10
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle13
after_count =gt 20
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 30
period =gt 1200
key =gt servicename_alarm_warning
Page of 37 71 Page of 37 71
add_tag =gt servicename_throttled
13 13 13 13 13 13 13 13 13
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 40
period =gt 1500
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
Enrich13 alerts13 with13 message13 detail13 Some13 of13 the13 third-shy‐party13 integraons13 benefit13 from13 having13 components13 of13 the13 raw13 log13 record13 sent13 in13 addion13 to13 the13 raw13 record13 or13 fields13 in13 the13 record13 edited13 transformed13 or13 translated13
Grok13 Logstash13 allows13 the13 log13 record13 to13 be13 broken13 up13 into13 logical13 secons13 and13 each13 secon13 stored13 in13 a13 new13 parameter13 using13 the13 grok13 parameter13 in13 the13 filter13 secon13 Here13 is13 an13 example13 grok13
13 grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
To13 build13 the13 grok13 statement13 Logstash13 offers13 an13 online13 tool13 at13 hcpsgrokdebugherokuappcom13
Translate13 If13 specific13 fields13 in13 the13 log13 record13 are13 coded13 (for13 example13 the13 Bluemix13 app13 name13 is13 oaen13 not13 sent13 in13 the13 syslog13 messages13 instead13 the13 GUID13 is13 sent13 which13 is13 not13 useful)13 then13 the13 filter13 secon13 translate13 plugin13 can13 be13 used13 to13 add13 new13 fields13 by13 holding13 the13 decoded13 versions13 of13 those13 values13
13 translate
field =gt data1
destination =gt data1
override =gt true
Page of 38 71 Page of 38 71
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
dc50f6e4-c052-436a-bf27-95ba392c4228 monverifyservicemanagement
0c677e82-5bfc-4e38-9155-e416249e3ee6 ms-catalog-postpyretic-pomiculture
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
However13 the13 translate13 plugin13 is13 not13 included13 with13 the13 Logstash13 installaon13 In13 the13 next13 secon13 I13 will13 13 describe13 how13 to13 add13 new13 plugins13 to13 the13 Logstash13 instance13
Add13 custom13 plugins13 to13 Logstash13 As13 an13 open13 source13 tool13 Logstash13 encourages13 users13 to13 develop13 content13 for13 all13 users13 to13 share13 Some13 of13 the13 plugins13 that13 were13 created13 by13 our13 community13 include13 a13 translaon13 plugin13 an13 output13 Slack13 plugin13 and13 an13 output13 syslog13 plugin13 Letrsquos13 look13 at13 how13 to13 add13 these13 plugins13 to13 your13 environment13
Filter13 translate13 plugin13 To13 add13 the13 translate13 plugin13
1 Download13 the13 plugin13
2 Copy13 the13 plugin13 file13 logstash-shy‐filter-shy‐translate-shy‐214gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-filter-translate 214
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Output13 Slack13 plugin13 To13 add13 the13 Slack13 plugin13
Page of 39 71 Page of 39 71
1 Download13 the13 plugin13 13
2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐slack-shy‐014gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-output-slack 014
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Output13 syslog13 plugin13 To13 add13 the13 syslog13 plugin13
1 Download13 the13 plugin13 13
2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐syslog-shy‐214gem13 to13 the13 ELK13 Virtual13 Server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-output-syslog 214
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Integrate13 Logstash13 with13 Slack13 To13 send13 generated13 alerts13 to13 Slack13 complete13 these13 steps13
1 Create13 a13 Slack13 integraon13 13
2 Use13 the13 Slack13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 Slack13 integraon13
113 Login13 to13 Slack13 and13 go13 to13 the13 Apps13 page13 13
Page of 40 71 Page of 40 71
13
313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13
13
A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13
13
Page of 42 71 Page of 42 71
The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13
13
The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13
13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
slack
url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5
channel =gt cloud-operations-mon
username =gt nick_cawood
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13
Page of 43 71 Page of 43 71
Page of 44 71 Page of 44 71
In13 Slack13 the13 alerts13 look13 like13 this13
13
Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13
1 Create13 a13 PagerDuty13 integraon13 13
2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13
1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13
13
2 Click13 Add13 New13 Service13
13
313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13
Page of 45 71 Page of 45 71
13
413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13
513 Click13 Add13 Service13
13
The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13
Page of 46 71 Page of 46 71
13
The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt be7eb1ea9c224018923e818d3e891d0b
incident_key =gt logstashhosttype
Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13
Page of 47 71 Page of 47 71
13
Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13
1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13
2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13
3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13
4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13
13 13 http
url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo
http_method =gt post
format =gt json
For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13
13 13 syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
Page of 48 71 Page of 48 71
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13
13
The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13
if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2
This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13
Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13
Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13
Logstash13 severity Shown13 in13 Omnibus13 as
alert Indeterminate
crical Warning
debug 7
emergency Clear
error Minor
informaonal 6
noce Crical
warning Major
Page of 49 71 Page of 49 71
13
Page of 50 71 Page of 50 71
Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13
input
tcp
port =gt 5000
type =gt syslog
filter
mutate
add_field =gt format =gt cf
grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
translate
field =gt data1
destination =gt data1
override =gt true
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
=== Check Alarm Criteria ===
Standard Key Words
if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
=== Alarm Exclusions ===
if [message] =~ (i)sometexthere
mutate remove_tag =gt servicename_alarm
Throttle alarms so we do not hit PD rate limits
Page of 52 71 Page of 52 71
if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 5
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle
after_count =gt 5
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 5
period =gt 1200
key =gt servicename_alarm_warning
add_tag =gt servicename_throttled
Page of 53 71 Page of 53 71
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
output
elasticsearch
hosts =gt localhost
stdout codec =gt rubydebug
if servicename_alarm_err in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
Page of 54 71 Page of 54 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 55 71 Page of 55 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
Page of 56 71 Page of 56 71
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert err message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_error in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
Page of 57 71 Page of 57 71
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 58 71 Page of 58 71
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 59 71 Page of 59 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert error message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
Page of 60 71 Page of 60 71
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
Page of 61 71 Page of 61 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 62 71 Page of 62 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt x
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert fatal message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
Page of 63 71 Page of 63 71
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
Page of 64 71 Page of 64 71
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 65 71 Page of 65 71
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt heavy_exclamation_mark
Page of 66 71 Page of 66 71
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert WARNING message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 67 71 Page of 67 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
Page of 68 71 Page of 68 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 69 71 Page of 69 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
Page of 70 71 Page of 70 71
Page of 71 71 Page of 71 71
13
The13 Bluemix13 cloud13 plaRorm13 offers13 a13 Public13 network13 and13 access13 to13 US13 South13 and13 UK13 environments13 From13 all13 these13 environments13 third-shy‐party13 cloud-shy‐based13 soluons13 are13 also13 accessible13
The13 ELK13 stack13 is13 installed13 in13 the13 Bluemix13 United13 Kingdom13 environment13 because13 this13 is13 the13 environment13 where13 Bluemix13 has13 virtual13 servers13 available13 for13 provisioning13 Other13 applicaons13 in13 the13 CASE13 project13 were13 also13 installed13 on13 virtual13 servers13 in13 the13 Bluemix13 United13 Kingdom13 region13
Bluemix13 apps13 were13 used13 to13 create13 a13 system13 to13 be13 monitored13 by13 ELK13 These13 apps13 were13 hosted13 in13 the13 Bluemix13 US13 South13 environment13 using13 Bluemix13 microservices13 which13 produce13 logs13 during13 normal13 operaon13 and13 during13 crashes13 and13 outages13
Through13 the13 Bluemix13 Cloud13 Foundry13 process13 the13 log13 records13 can13 be13 broadcast13 from13 the13 Bluemix13 apps13 so13 Logstash13 can13 receive13 them13 and13 send13 them13 to13 Elascsearch13 within13 the13 ELK13 stack13 or13 other13 virtual13 servers13 (hosng13 other13 IBM13 or13 third-shy‐party13 products)13 in13 the13 Bluemix13 United13 Kingdom13 environment13 or13 to13 third-shy‐party13 applicaons13 in13 the13 public13 cloud13 (Pager13 Duty13 Slack13 etc)13
Crea0ng13 a13 virtual13 server13 in13 Bluemix13 With13 Bluemix13 you13 can13 provision13 a13 virtual13 server13 in13 the13 cloud13 to13 install13 standalone13 or13 integrated13 system13 components13 Not13 all13 Bluemix13 regions13 offer13 this13 service13 all13 the13 me13 due13 to13 beta-shy‐project13 constraints13 In13 this13 arclersquos13 examples13 virtual13 servers13 were13 allowed13 to13 be13 provisioned13 in13 the13 United13 Kingdom13 region13 For13 more13 informaon13 read13 how13 to13 provision13 a13 Bluemix13 virtual13 server13 including13 some13 simple13 setup13 steps13 for13 ease13 of13 use13
Page of 7 71 Page of 7 71
Create13 an13 SSH13 key13 To13 access13 your13 virtual13 server13 once13 it13 is13 provisioned13 allocate13 an13 SSH13 key13 to13 the13 virtual13 server13
To13 generate13 a13 publicprivate13 RSA13 key13 pair13 follow13 these13 steps13
1 In13 a13 UNIX13 or13 Linuxreg13 window13 run13 the13 following13 command13 specifying13 a13 ltuniquekeynamegt13 of13 your13 your13 choice13 and13 a13 password13 or13 passphrase13 (memorize13 the13 password)$13 ssh-shy‐keygen13 -shy‐t13 rsa13 -shy‐f13 ltuniquekeynamegt13
2 Enter13 the13 passphrase13 (empty13 for13 no13 passphrase)13
3 Enter13 the13 same13 passphrase13 again13
Your13 idenficaon13 has13 been13 saved13 in13 ltuniquekeynamegt13
Your13 public13 key13 has13 been13 saved13 in13 ltuniquekeynamegtpub13
The13 key13 fingerprint13 is13
d6268464506525930ce4632c693c203c13 ltusergtltservergt13
The13 keys13 randomart13 image13 is13
+--[ RSA 2048]----+
|o ++=o |
| E o ooo |
| |
| + o |
| S o |
| o |
| |
| |
| |
+-----------------+
Two13 keys13 are13 created13 public13 and13 private13 The13 public13 key13 contents13 are13 used13 to13 create13 a13 virtual13 server13 with13 a13 public13 IP13 address13 The13 private13 key13 is13 used13 to13 access13 the13 virtual13 server13 from13 a13 client13 machine13
Page of 8 71 Page of 8 71
Provisioning13 a13 virtual13 server13 1 A13 Bluemix13 account13 is13 required13 for13 this13 process13 Sign13 up13 here13 and13 log13 in13 Make13 sure13 your13
environment13 is13 correct13 For13 example13 if13 you13 are13 in13 the13 United13 Kingdom13 select13 the13 United13 Kingdom13 region13 in13 the13 Bluemix13 console13
13
13
2 Add13 a13 virtual13 server13 to13 your13 Bluemix13 environment13 Click13 Run13 Virtual13 Servers13
13
3 Select13 an13 OS13 image13 from13 the13 EXISTING13 list13 For13 ELK13 select13 Centos13 OS13 (use13 latest13 available13 version)13
Page of 9 71 Page of 9 71
4 Name13 the13 virtual13 server13
5 Make13 sure13 Assign13 public13 IP13 address13 is13 checked13
13
6 Click13 Add13 Key13 to13 add13 the13 SSH13 key13 you13 created13 previously13 Copy13 the13 contents13 of13 the13 public13 key13 into13 the13 Public13 Key13 to13 import13 field13
Page of 10 71 Page of 10 71
13
7 Click13 OK13 Then13 click13 CREATE13
The13 virtual13 server13 is13 provisioned13 and13 added13 to13 the13 Bluemix13 dashboard13 under13 VIRTUAL13 SERVERS13
Page of 11 71 Page of 11 71
13
Set13 up13 the13 virtual13 server13
Log13 on13 to13 the13 virtual13 server13 Now13 you13 can13 log13 on13 to13 the13 virtual13 server13 and13 add13 new13 users13 and13 programs13 to13 make13 it13 easier13 to13 access13 the13 virtual13 server13
To13 access13 the13 server13 go13 to13 the13 UNIX13 or13 Linux13 window13 that13 you13 used13 to13 create13 the13 SSH13 key13 In13 the13 directory13 where13 the13 ltuniquekeynamegt13 files13 exist13 run13 the13 following13 command13 referencing13 the13 SSH13 keyname13 the13 public13 IP13 address13 Bluemix13 allocated13 for13 the13 virtual13 server13 and13 the13 password13 for13 the13 SSH13 key13
[userserver ~]$ ssh -i ltuniquekeynamegt ibmcloudltpublicIPaddressgt
Enter13 the13 passphrase13 for13 the13 key13 ltuniquekeynamegt13
Last login Tue Jul 5 125546 2016 from 108-210-96-231lightspeedrlghncsbcglobalnet
[ibmcloudelkforcase ~]$
ibmcloud13 is13 the13 default13 user13 for13 the13 Bluemix13 virtual13 servers13 and13 allows13 SSH13 to13 access13 the13 server13
Acces13 the13 root13 user13 To13 access13 the13 root13 user13 log13 on13 as13 ibmcloud13 user13 and13 run13 the13 following13 command13
[ibmcloudelkforcase ~]$ sudo bash
[rootelkforcase ibmcloud]
The13 root13 user13 can13 be13 used13 to13 add13 programs13 to13 the13 server13 and13 create13 and13 edit13 users13 as13 required13 to13 access13 applicaons13 installed13 on13 the13 virtual13 server13
Page of 12 71 Page of 12 71
Copy13 files13 to13 the13 virtual13 server13 To13 move13 installaon13 files13 to13 the13 server13 go13 to13 the13 UNIX13 or13 Linux13 window13 you13 used13 to13 create13 the13 SSH13 key13 and13 the13 directory13 where13 the13 ltuniquekeynamegt13 files13 exist13 Move13 the13 installaon13 files13 to13 the13 same13 locaon13
Run13 the13 following13 command13 referencing13 the13 SSH13 keyname13 the13 public13 IP13 address13 Bluemix13 allocated13 for13 the13 virtual13 server13 and13 the13 password13 for13 the13 SSH13 key13
[userserver ~]$ scp ndashi ltuniquekeynamegt ltINSTALLATIONFILENAMEgt ibmcloudltpublicIPaddressgthomeibmcloud
Enter13 the13 passphrase13 for13 key13 cloudkey313
This13 copies13 the13 file13 using13 the13 ibmcloud13 user13 to13 the13 homeibmcloud13 directory13 on13 the13 virtual13 server13
Install13 and13 access13 the13 virtual13 server13 via13 VNC13 server13 Using13 the13 ssh13 command13 to13 access13 the13 virtual13 server13 is13 not13 the13 most13 stable13 method13 and13 does13 not13 offer13 X-shy‐Windows13 support13 Therefore13 we13 suggest13 that13 you13 use13 a13 VNC13 client13 to13 access13 the13 virtual13 server13
Installing13 a13 VNC13 client13 should13 be13 done13 as13 the13 root13 user13 Use13 the13 yum13 command13 as13 follows13
[ibmcloudelkforcase ~]$ yum groupinstall GNOME Desktop
[ibmcloudelkforcase ~]$ yum install tigervnc-server
These13 commands13 first13 install13 a13 desktop13 style13 OS13 user13 interface13 and13 then13 a13 VNC13 client13 to13 remotely13 access13 the13 desktop13 style13 OS13
Then13 as13 either13 the13 root13 user13 or13 another13 user13 (example13 below13 uses13 elk)13 run13 the13 following13 command13 to13 create13 the13 VNC13 client13 connecon13
[elkelkforcase ~]$ vncserver
New elkforcase2 (elk) desktop is elkforcase2
Starting applications specified in homeelkvncxstartup
Log file is homeelkvncelkforcase2log
[elkelkforcase ~]$
Install13 a13 VNC13 client13 on13 the13 client13 machine13 used13 to13 access13 the13 Internet13 and13 (in13 the13 example13 above)13 use13 the13 command13 ltpublicIPadddressgt213 and13 the13 password13 for13 ELK13 to13 access13 the13 virtual13 server13
Using13 a13 VNC13 client13 means13 that13 you13 can13 leave13 long13 duraon13 installaons13 running13 without13 fear13 of13 the13 terminal13 losing13 connecon13 and13 the13 installaons13 failing13
Install13 the13 ELK13 stack13 Download13 the13 source13 soaware13 code13 for13 the13 ELK13 stack13 from13 the13 ELK13 website13 13
At13 the13 me13 of13 wring13 the13 downloaded13 soaware13 files13 were13
Page of 13 71 Page of 13 71
13
Transfer13 the13 files13 to13 the13 virtual13 server13 (using13 the13 scp13 method13 detailed13 above)13
Install13 Elas0csearch13 As13 a13 root13 user13 create13 a13 non-shy‐root13 user13 with13 the13 adduser13 command13
Create13 a13 directory13 to13 install13 Elascsearch13 into13 and13 untar13 the13 contents13 of13 the13 Elascsearch13 install13 file13 into13 that13 directory13
[elkelkforcase]$ cd home
[elkelkforcase]$ mkdir elasticsearch
[elkelkforcase]$ cp homeibmcloudelasticsearch-233targz homeelasticsearch
[elkelkforcase]$ cd elasticsearch
[elkelkforcase]$ gunzip elasticsearch-233targz
[elkelkforcase]$ untar elasticsearch-233tar
Elascsearch13 is13 now13 installed13
Start13 Elas0csearch13 To13 start13 Elascsearch13 go13 to13 the13 bin13 directory13 as13 the13 non-shy‐root13 user13 when13 you13 installed13 it13 and13 run13 the13 elascsearch13 executable13
[elkelkforcase]$ cd homeelasticsearchelasticsearch-233bin
[elkelkforcase]$ elasticsearch amp
[2] 22211
[elkelkforcase]$ [2016-07-05 133857730][INFO ][node] [Arsenal] version[233] pid[22211] build[218bdf12016-05-17T154004Z]
[2016-07-05 133857740][INFO ][node] [Arsenal] initializing
[2016-07-05 133858984][INFO ][plugins] [Arsenal] modules [reindex lang-expression lang-groovy] plugins [] sites []
[2016-07-05 133859146][INFO ][env] [Arsenal] using [1] data paths mounts [[ (rootfs)]] net usable_space [155gb] net total_space [199gb] spins [unknown] types [rootfs]
[2016-07-05 133859146][INFO ][env] [Arsenal] heap size [10156mb] compressed ordinary object pointers [true]
Page of 14 71 Page of 14 71
[2016-07-05 133859147][WARN ][env] [Arsenal] max file descriptors [4096] for elasticsearch process likely too low consider increasing to at least [65536]
[2016-07-05 133903526][INFO ][node] [Arsenal] initialized
[2016-07-05 133903526][INFO ][node] [Arsenal] starting
To13 test13 that13 Elascsearch13 is13 running13 correctly13 open13 the13 browser13 in13 the13 VNC13 client13 and13 run13 localhost920013
13
Install13 Kibana13 ELK13 recommends13 installing13 and13 running13 Kibana13 as13 root13 13
1 As13 root13 user13 create13 an13 installaon13 directory13 and13 untar13 the13 installaon13 file13
[rootelkforcase]$ cd home
[rootelkforcase]$ mkdir kibana
[rootelkforcase]$ cp homeibmcloudkibana-451-linux-x64targz homekibana
[rootelkforcase]$ cd kibana
[rootelkforcase]$ gunzip kibana-451-linux-x64targz
[rootelkforcase]$ untar kibana-451-linux-x64tar
2 Modify13 the13 manifestyml13 configuraon13 file13 Go13 to13 the13 kibana-shy‐451-shy‐linux-shy‐x6413 directory13
[rootelkforcase]$ cd homekibanakibana-451-linux-x64
3 Edit13 the13 manifestyml13 file13 so13 the13 contents13 are13
applicaons13
-shy‐ name13 kibana-shy‐in-shy‐bluemix13
memory13 128M13
host13 kibana-shy‐in-shy‐bluemix13
buildpack13 hcpsgithubcomcloudfoundry-shy‐communitynginx-shy‐buildpackgit13
Page of 15 71 Page of 15 71
4 Edit13 the13 configkibanayml13 file13 so13 the13 contents13 are13
Kibana is served by a back end server This controls which port to use
port 5601
The host to bind the server to
host 0000
If you are running kibana behind a proxy and want to mount it at a path
specify that path here The basePath cant end in a slash
serverbasePath
The maximum payload size in bytes on incoming server requests
servermaxPayloadBytes 1048576
The Elasticsearch instance to use for all your queries
elasticsearchurl httplocalhost9200
preserve_elasticsearch_host true will send the hostname specified in `elasticsearch` If you set it to false
then the host you use to connect to this Kibana instance will be sent
elasticsearchpreserveHost true
Kibana uses an index in Elasticsearch to store saved searches visualizations
and dashboards It will create a new index if it doesnt already exist
kibanaindex kibana
The default application to load
kibanadefaultAppId discover
If your Elasticsearch is protected with basic auth these are the user credentials
used by the Kibana server to perform maintenance on the kibana_index at startup Your
Kibana users will still need to authenticate with Elasticsearch (which is proxied through
the Kibana server)
Page of 16 71 Page of 16 71
elasticsearchusername user
elasticsearchpassword pass
SSL for outgoing requests from the Kibana Server to the browser (PEM formatted)
serversslcert pathtoyourservercrt
serversslkey pathtoyourserverkey
Optional setting to validate that your Elasticsearch backend uses the same key files (PEM formatted)
elasticsearchsslcert pathtoyourclientcrt
elasticsearchsslkey pathtoyourclientkey
If you need to provide a CA certificate for your Elasticsearch instance put
the path of the pem file here
elasticsearchsslca pathtoyourCApem
Set to false to have a complete disregard for the validity of the SSL
certificate
elasticsearchsslverify true
Time in milliseconds to wait for elasticsearch to respond to pings defaults to
request_timeout setting
elasticsearchpingTimeout 1500
Time in milliseconds to wait for responses from the back end or elasticsearch
This must be gt 0
elasticsearchrequestTimeout 30000
Time in milliseconds for Elasticsearch to wait for responses from shards
Set to 0 to disable
elasticsearchshardTimeout 0
Time in milliseconds to wait for Elasticsearch at Kibana startup before retrying
elasticsearchstartupTimeout 5000
elasticsearchstartupTimeout 30000
Page of 17 71 Page of 17 71
Set the path to where you would like the process id file to be created
pidfile varrunkibanapid
If you would like to send the log output to a file you can set the path below
loggingdest stdout
Set this to true to suppress all logging output
loggingsilent false
Set this to true to suppress all logging output except for error messages
loggingquiet false
Set this to true to log all events including system usage information and all requests
loggingverbose false
Kibana13 is13 now13 installed13
Start13 Kibana13 To13 start13 Kibana13 go13 to13 the13 bin13 directory13 and13 run13 the13 kibana13 executable13
[rootelkforcase]$
[rootelkforcase]$ cd homekibanakibana-451-linux-x64bin
[rootelkforcase]$ kibana amp
[1] 22283
[rootelkforcase] log [095900542] [info][status][pluginkibana] Status changed from uninitialized to green - Ready
log [095900600] [info][status][pluginelasticsearch] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [095900689] [info][status][pluginkbn_vislib_vis_types] Status changed from uninitialized to green - Ready
log [095900717] [info][status][pluginmarkdown_vis] Status changed from uninitialized to green - Ready
log [095900721] [info][status][pluginmetric_vis] Status changed from uninitialized to green - Ready
log [095900814] [info][status][pluginspyModes] Status changed from uninitialized to green - Ready
log [095900934] [info][status][pluginstatusPage] Status changed from uninitialized to green - Ready
Page of 18 71 Page of 18 71
log [095900941] [info][status][plugintable_vis] Status changed from uninitialized to green - Ready
log [095901067] [info][listening] Server running at http00005601
log [095906083] [info][status][pluginelasticsearch] Status changed from yellow to yellow - No existing Kibana index found
log [095908942] [info][status][pluginelasticsearch] Status changed from yellow to green - Kibana index ready
Kibana13 is13 now13 started13
Broadcast13 Bluemix13 app13 log13 records13 Bluemix13 apps13 (which13 are13 shown13 in13 the13 US13 South13 Bluemix13 region13 of13 the13 System13 Architecture13 diagram)13 will13 produce13 logs13 in13 a13 syslog13 rfc542413 format13 using13 a13 Bluemix13 tool13 called13 Loggregator13
For13 Logstash13 to13 receive13 these13 logs13 each13 Bluemix13 app13 needs13 to13 broadcast13 these13 logs13 To13 do13 that13 run13 the13 following13 commands13 on13 each13 Bluemix13 app13 using13 the13 public13 IP13 address13 of13 the13 ELK13 virtual13 server13 the13 ELK13 tag13 to13 trace13 the13 process13 and13 insert13 the13 name13 of13 each13 app13 in13 turn13
cf cups elk -l syslogltpublicIPaddress5000 cf bind-service ltmyappgt elk
cf restage ltmyappgt
Install13 Logstash13 To13 install13 Logstash13 as13 root13 user13 create13 a13 directory13 for13 the13 installaon13 and13 untar13 the13 installaon13 file13 there13
13 13 [rootelkforcase]$ cd home
[rootelkforcase]$ mkdir logstash
[rootelkforcase]$ cp homeibmcloudlogstash-233targz homelogstash
[rootelkforcase]$ cd logstash
[rootelkforcase]$ gunzip logstash-233targz
[rootelkforcase]$ untar logstash-233tar
Write13 Logstash13 configura0on13 Logstash13 requires13 a13 configuraon13 file13 to13 be13 wricen13 to13 receive13 transform13 and13 output13 the13 log13 records13 to13 Elascsearch13 This13 secon13 explains13 the13 basic13 requirements13 of13 processing13 Bluemix13 app13 log13 records13 into13 Elascsearch13 (more13 complex13 configuraons13 are13 detailed13 later13 in13 this13 document13 for13 the13 integraons13 with13 third-shy‐party13 tools)13
Create13 a13 conf13 file13 as13 root13 user13
[rootelkforcase]$ cd homelogstashlogstash-233
[rootelkforcase]$ mkdir config
Page of 19 71 Page of 19 71
[rootelkforcase]$ cd config
[rootelkforcase]$ touch loggregatorconf
Edit13 the13 loggregator13 file13 with13 the13 following13 contents13
13 Input13 secon13 defines13 which13 log13 records13 to13 receive13
13 port13 and13 type13 reflect13 method13 selected13 for13 Cloud13 Foundry13 to13 broadcast13 the13 records13
input
tcp
port =gt 5000
type =gt syslog
13 Filter13 secon13 ndash13 transforms13 the13 data13
filter
Add a field to identify the records as Cloud Foundry based
mutate
add_field =gt format =gt cf
13 Output13 secon13 ndash13 sends13 the13 data13 to13 its13 desnaon(s)13
output
13 Use13 the13 elascsearch13 plugin13 to13 send13 the13 data13 to13 the13 elascsearch13 instance13
13 elasticsearch
hosts =gt localhost
stdout codec =gt rubydebug
13 End13 of13 Config13
Save13 the13 file13 Verify13 the13 format13 of13 the13 file13 is13 acceptable13 by13 running13 the13 following13 commands13
[rootelkforcase]$ cd homelogstashlogstash-233
[rootelkforcase]$ binlogstash agent --configtest -f configloggregatorconf
Page of 20 71 Page of 20 71
Configuraon13 is13 OK13
Start13 Logstash13 Aaer13 you13 have13 wricen13 and13 verified13 the13 Logstash13 configuraon13 you13 can13 start13 Logstash13 to13 process13 records13 from13 the13 Bluemix13 apps13 (via13 Cloud13 Foundry)13 Run13 the13 following13 as13 root13 user13
[rootelkforcase]$ cd homelogstashlogstash-233
[rootelkforcase]$ binlogstash agent -f configloggregatorconf
Logstash13 is13 now13 started13
Monitor13 Logstash13 processing13 Once13 Logstash13 is13 running13 and13 log13 records13 are13 being13 produced13 by13 the13 Bluemix13 apps13 you13 can13 monitor13 progress13 in13 the13 same13 window13 that13 was13 used13 to13 start13 Logstash13 Output13 is13 similar13 to13 the13 following13
13
The13 log13 records13 being13 broadcast13 from13 the13 Bluemix13 apps13 are13 now13 sent13 to13 Elascsearch13 from13 the13 Logstash13 output13 configuraon13 Records13 are13 sent13 in13 real13 me13 If13 Logstash13 is13 not13 running13 while13 log13 records13 are13 being13 produced13 then13 these13 records13 will13 not13 be13 processed13 by13 Logstash13 and13 will13 not13 be13 stored13 in13 Elascsearch13
Page of 21 71 Page of 21 71
Use13 ELK13 The13 log13 records13 from13 the13 Bluemix13 apps13 are13 now13 being13 stored13 and13 indexed13 by13 the13 Elascsearch13 component13 This13 data13 can13 be13 accessed13 via13 the13 Kibana13 applicaon13
Kibana13 Discover13 page13 Using13 a13 web13 browser13 either13 directly13 from13 a13 client13 or13 using13 the13 VNC13 client13 access13 Kibana13 using13 the13 publicIPaddress13 of13 the13 ELK13 virtual13 server13
hcpltpublicIPaddressgt560113
No13 userpassword13 credenals13 are13 required13
Kibana13 defaults13 to13 a13 search13 of13 all13 records13 ()13 in13 the13 last13 1513 minutes13 and13 returns13 the13 results13
13
More13 complex13 searches13 are13 possible13 by13 changing13 the13 me13 filter13 and13 supplying13 search13 queries13
Change13 the13 0me13 filter13 To13 change13 the13 me13 ranges13 click13 the13 clock13 symbol13 in13 the13 top-shy‐right13 of13 the13 Kibana13 Discover13 page13
13
You13 will13 see13 the13 following13 opons13
13
Page of 22 71 Page of 22 71
A13 variety13 of13 quick13 me13 ranges13 can13 be13 selected13 or13 you13 can13 switch13 to13 relave13 or13 absolute13 me13 ranges13
Quick13 me13 ranges13 allow13 switching13 between13 last13 X13 minutes13 to13 last13 Y13 hours13 or13 Z13 days13 etc13
Relave13 me13 ranges13 allow13 more13 specific13 versions13 of13 the13 Quick13 opons13 from13 specific13 minuteshoursdays13 to13 now13
13
Absolute13 me13 ranges13 specify13 the13 start13 and13 end13 mes13 for13 the13 searches13
13
Custom13 search13 queries13 You13 can13 use13 the13 text13 window13 of13 the13 Discover13 page13 to13 search13 for13 matches13 within13 the13 log13 records13 You13 can13 use13 Apache13 Lucene13 or13 Elascsearch13 query13 DSL13 syntax13
13
Page of 23 71 Page of 23 71
Saved13 searches13 You13 can13 save13 any13 custom13 search13 syntax13 and13 choice13 of13 me13 range13 for13 future13 ease13 of13 use13 Any13 relave13 or13 absolute13 me13 frame13 is13 saved13 as13 part13 of13 the13 search13 13
To13 save13 a13 search13
1 Set13 up13 and13 run13 the13 required13 syntax13 and13 me13 frame13
13
2 Click13 the13 Save13 Search13 icon13
13
3 Type13 in13 a13 name13 for13 the13 saved13 search13
Page of 24 71 Page of 24 71
13
7 To13 delete13 or13 update13 saved13 searches13 click13 13 manage13 saved13 searches13
13
Use13 the13 edit13 icons13 to13 make13 any13 changes13
Page of 26 71 Page of 26 71
13
Build13 simple13 Kibana13 histograms13 Kibana13 allows13 you13 to13 build13 charts13 and13 dashboards13 based13 on13 the13 search13 capability13 offered13 by13 the13 Discover13 page13 Saved13 searches13 can13 be13 used13 as13 the13 basis13 for13 charts13 and13 dashboards13 built13 using13 those13 charts13 To13 build13 a13 simple13 histogram13
1 Select13 the13 Visualize13 page13
13
2 Select13 Ver0cal13 Bar13 Chart13
13
3 To13 use13 a13 saved13 search13 click13 From13 a13 saved13 search13
Page of 27 71 Page of 27 71
13
4 Select13 a13 previously13 saved13 search13
13
5 Select13 to13 edit13 the13 x-shy‐axis
6 Edit13 the13 fields13 with13 informaon13 about13 the13 aggregaon13 field13 interval13 and13 custom13 label13
Page of 28 71 Page of 28 71
13
Page of 29 71 Page of 29 71
7 Click13 the13 green13 arrow13 The13 histogram13 is13 created13 and13 run13 based13 on13 the13 saved13 search
13
8 To13 save13 the13 histogram13 click13 the13 Save13 Visualiza0on13 icon13 and13 add13 a13 tle13
13
13
9 Click13 Save13
10 To13 view13 the13 visualisaon13 click13 the13 Dashboard13 page13
Page of 30 71 Page of 30 71
13
11 Click13 the13 Add13 Visualiza0on13 icon13
13
12 Select13 the13 visiualizaon13 that13 you13 just13 created13
13
Page of 31 71 Page of 31 71
13
13 To13 save13 the13 dashboard13 click13 the13 Save13 Dashboard13 icon13
Page of 32 71 Page of 32 71
13
14 Add13 a13 tle13 and13 select13 whether13 to13 store13 the13 selected13 me13 frame13 with13 the13 dashboard13
13
15 Click13 Save13
16 You13 can13 edit13 the13 shape13 and13 number13 of13 charts13 and13 save13 the13 dashboard13
17 To13 re-shy‐run13 the13 dashboard13 select13 the13 Dashboard13 page13 and13 click13 the13 Load13 Saved13 Dashboard13 icon13
Page of 33 71 Page of 33 71
13
18 Select13 the13 saved13 dashboard13 from13 the13 list13
13
19 The13 dashboard13 runs13
13
Page of 34 71 Page of 34 71
Page of 35 71 Page of 35 71
Aler0ng13 with13 ELK13 ELK13 examines13 the13 log13 records13 in13 process13 to13 send13 alerts13 to13 third-shy‐party13 alerng13 event13 management13 and13 social13 media13 applicaons13 As13 shown13 previously13 in13 the13 ELK13 architecture13 the13 ELK13 strategy13 for13 alerng13 is13 to13 send13 alerts13 from13 the13 Logstash13 component13 directly13 Therefore13 log13 records13 and13 alerts13 are13 sent13 to13 both13 Elascsearch13 and13 third-shy‐party13 tools13 at13 the13 same13 instant13 using13 the13 output13 secon13 of13 the13 Logstash13 configuraon13 Any13 alerts13 are13 generated13 during13 the13 filter13 secon13 processing13 of13 the13 Logstash13 configuraon13
Create13 alerts13 in13 Logstash13 The13 Logstash13 filter13 secon13 is13 used13 to13 trigger13 alerts13 based13 on13 the13 log13 records13 being13 processed13 For13 example13 log13 records13 can13 be13 checked13 for13 errors13 warnings13 crashed13 or13 stopped13 messages13 and13 generate13 an13 alert13
Here13 is13 an13 example13 Logstash13 filter13 code13
13 if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
The13 specific13 tags13 are13 set13 against13 the13 log13 records13 to13 trigger13 the13 alert13 later13 in13 the13 configuraon13 in13 the13 output13 secon13
Control13 the13 number13 of13 generated13 alerts13 using13 the13 Logstash13 throcle13 plugin13 so13 that13 either13 a13 lot13 of13 instances13 in13 a13 fixed13 me13 period13 are13 required13 before13 an13 alert13 is13 sent13 or13 a13 specific13 number13 of13 instances13 triggers13 the13 alert13
Page of 36 71 Page of 36 71
Edit13 the13 plugin13 parameters13 to13 meet13 system13 requirements13
13 13 13 13 13 if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 10
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle13
after_count =gt 20
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 30
period =gt 1200
key =gt servicename_alarm_warning
Page of 37 71 Page of 37 71
add_tag =gt servicename_throttled
13 13 13 13 13 13 13 13 13
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 40
period =gt 1500
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
Enrich13 alerts13 with13 message13 detail13 Some13 of13 the13 third-shy‐party13 integraons13 benefit13 from13 having13 components13 of13 the13 raw13 log13 record13 sent13 in13 addion13 to13 the13 raw13 record13 or13 fields13 in13 the13 record13 edited13 transformed13 or13 translated13
Grok13 Logstash13 allows13 the13 log13 record13 to13 be13 broken13 up13 into13 logical13 secons13 and13 each13 secon13 stored13 in13 a13 new13 parameter13 using13 the13 grok13 parameter13 in13 the13 filter13 secon13 Here13 is13 an13 example13 grok13
13 grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
To13 build13 the13 grok13 statement13 Logstash13 offers13 an13 online13 tool13 at13 hcpsgrokdebugherokuappcom13
Translate13 If13 specific13 fields13 in13 the13 log13 record13 are13 coded13 (for13 example13 the13 Bluemix13 app13 name13 is13 oaen13 not13 sent13 in13 the13 syslog13 messages13 instead13 the13 GUID13 is13 sent13 which13 is13 not13 useful)13 then13 the13 filter13 secon13 translate13 plugin13 can13 be13 used13 to13 add13 new13 fields13 by13 holding13 the13 decoded13 versions13 of13 those13 values13
13 translate
field =gt data1
destination =gt data1
override =gt true
Page of 38 71 Page of 38 71
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
dc50f6e4-c052-436a-bf27-95ba392c4228 monverifyservicemanagement
0c677e82-5bfc-4e38-9155-e416249e3ee6 ms-catalog-postpyretic-pomiculture
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
However13 the13 translate13 plugin13 is13 not13 included13 with13 the13 Logstash13 installaon13 In13 the13 next13 secon13 I13 will13 13 describe13 how13 to13 add13 new13 plugins13 to13 the13 Logstash13 instance13
Add13 custom13 plugins13 to13 Logstash13 As13 an13 open13 source13 tool13 Logstash13 encourages13 users13 to13 develop13 content13 for13 all13 users13 to13 share13 Some13 of13 the13 plugins13 that13 were13 created13 by13 our13 community13 include13 a13 translaon13 plugin13 an13 output13 Slack13 plugin13 and13 an13 output13 syslog13 plugin13 Letrsquos13 look13 at13 how13 to13 add13 these13 plugins13 to13 your13 environment13
Filter13 translate13 plugin13 To13 add13 the13 translate13 plugin13
1 Download13 the13 plugin13
2 Copy13 the13 plugin13 file13 logstash-shy‐filter-shy‐translate-shy‐214gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-filter-translate 214
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Output13 Slack13 plugin13 To13 add13 the13 Slack13 plugin13
Page of 39 71 Page of 39 71
1 Download13 the13 plugin13 13
2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐slack-shy‐014gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-output-slack 014
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Output13 syslog13 plugin13 To13 add13 the13 syslog13 plugin13
1 Download13 the13 plugin13 13
2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐syslog-shy‐214gem13 to13 the13 ELK13 Virtual13 Server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-output-syslog 214
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Integrate13 Logstash13 with13 Slack13 To13 send13 generated13 alerts13 to13 Slack13 complete13 these13 steps13
1 Create13 a13 Slack13 integraon13 13
2 Use13 the13 Slack13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 Slack13 integraon13
113 Login13 to13 Slack13 and13 go13 to13 the13 Apps13 page13 13
Page of 40 71 Page of 40 71
13
313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13
13
A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13
13
Page of 42 71 Page of 42 71
The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13
13
The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13
13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
slack
url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5
channel =gt cloud-operations-mon
username =gt nick_cawood
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13
Page of 43 71 Page of 43 71
Page of 44 71 Page of 44 71
In13 Slack13 the13 alerts13 look13 like13 this13
13
Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13
1 Create13 a13 PagerDuty13 integraon13 13
2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13
1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13
13
2 Click13 Add13 New13 Service13
13
313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13
Page of 45 71 Page of 45 71
13
413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13
513 Click13 Add13 Service13
13
The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13
Page of 46 71 Page of 46 71
13
The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt be7eb1ea9c224018923e818d3e891d0b
incident_key =gt logstashhosttype
Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13
Page of 47 71 Page of 47 71
13
Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13
1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13
2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13
3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13
4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13
13 13 http
url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo
http_method =gt post
format =gt json
For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13
13 13 syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
Page of 48 71 Page of 48 71
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13
13
The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13
if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2
This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13
Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13
Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13
Logstash13 severity Shown13 in13 Omnibus13 as
alert Indeterminate
crical Warning
debug 7
emergency Clear
error Minor
informaonal 6
noce Crical
warning Major
Page of 49 71 Page of 49 71
13
Page of 50 71 Page of 50 71
Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13
input
tcp
port =gt 5000
type =gt syslog
filter
mutate
add_field =gt format =gt cf
grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
translate
field =gt data1
destination =gt data1
override =gt true
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
=== Check Alarm Criteria ===
Standard Key Words
if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
=== Alarm Exclusions ===
if [message] =~ (i)sometexthere
mutate remove_tag =gt servicename_alarm
Throttle alarms so we do not hit PD rate limits
Page of 52 71 Page of 52 71
if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 5
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle
after_count =gt 5
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 5
period =gt 1200
key =gt servicename_alarm_warning
add_tag =gt servicename_throttled
Page of 53 71 Page of 53 71
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
output
elasticsearch
hosts =gt localhost
stdout codec =gt rubydebug
if servicename_alarm_err in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
Page of 54 71 Page of 54 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 55 71 Page of 55 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
Page of 56 71 Page of 56 71
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert err message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_error in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
Page of 57 71 Page of 57 71
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 58 71 Page of 58 71
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 59 71 Page of 59 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert error message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
Page of 60 71 Page of 60 71
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
Page of 61 71 Page of 61 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 62 71 Page of 62 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt x
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert fatal message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
Page of 63 71 Page of 63 71
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
Page of 64 71 Page of 64 71
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 65 71 Page of 65 71
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt heavy_exclamation_mark
Page of 66 71 Page of 66 71
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert WARNING message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 67 71 Page of 67 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
Page of 68 71 Page of 68 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 69 71 Page of 69 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
Page of 70 71 Page of 70 71
Page of 71 71 Page of 71 71
Create13 an13 SSH13 key13 To13 access13 your13 virtual13 server13 once13 it13 is13 provisioned13 allocate13 an13 SSH13 key13 to13 the13 virtual13 server13
To13 generate13 a13 publicprivate13 RSA13 key13 pair13 follow13 these13 steps13
1 In13 a13 UNIX13 or13 Linuxreg13 window13 run13 the13 following13 command13 specifying13 a13 ltuniquekeynamegt13 of13 your13 your13 choice13 and13 a13 password13 or13 passphrase13 (memorize13 the13 password)$13 ssh-shy‐keygen13 -shy‐t13 rsa13 -shy‐f13 ltuniquekeynamegt13
2 Enter13 the13 passphrase13 (empty13 for13 no13 passphrase)13
3 Enter13 the13 same13 passphrase13 again13
Your13 idenficaon13 has13 been13 saved13 in13 ltuniquekeynamegt13
Your13 public13 key13 has13 been13 saved13 in13 ltuniquekeynamegtpub13
The13 key13 fingerprint13 is13
d6268464506525930ce4632c693c203c13 ltusergtltservergt13
The13 keys13 randomart13 image13 is13
+--[ RSA 2048]----+
|o ++=o |
| E o ooo |
| |
| + o |
| S o |
| o |
| |
| |
| |
+-----------------+
Two13 keys13 are13 created13 public13 and13 private13 The13 public13 key13 contents13 are13 used13 to13 create13 a13 virtual13 server13 with13 a13 public13 IP13 address13 The13 private13 key13 is13 used13 to13 access13 the13 virtual13 server13 from13 a13 client13 machine13
Page of 8 71 Page of 8 71
Provisioning13 a13 virtual13 server13 1 A13 Bluemix13 account13 is13 required13 for13 this13 process13 Sign13 up13 here13 and13 log13 in13 Make13 sure13 your13
environment13 is13 correct13 For13 example13 if13 you13 are13 in13 the13 United13 Kingdom13 select13 the13 United13 Kingdom13 region13 in13 the13 Bluemix13 console13
13
13
2 Add13 a13 virtual13 server13 to13 your13 Bluemix13 environment13 Click13 Run13 Virtual13 Servers13
13
3 Select13 an13 OS13 image13 from13 the13 EXISTING13 list13 For13 ELK13 select13 Centos13 OS13 (use13 latest13 available13 version)13
Page of 9 71 Page of 9 71
4 Name13 the13 virtual13 server13
5 Make13 sure13 Assign13 public13 IP13 address13 is13 checked13
13
6 Click13 Add13 Key13 to13 add13 the13 SSH13 key13 you13 created13 previously13 Copy13 the13 contents13 of13 the13 public13 key13 into13 the13 Public13 Key13 to13 import13 field13
Page of 10 71 Page of 10 71
13
7 Click13 OK13 Then13 click13 CREATE13
The13 virtual13 server13 is13 provisioned13 and13 added13 to13 the13 Bluemix13 dashboard13 under13 VIRTUAL13 SERVERS13
Page of 11 71 Page of 11 71
13
Set13 up13 the13 virtual13 server13
Log13 on13 to13 the13 virtual13 server13 Now13 you13 can13 log13 on13 to13 the13 virtual13 server13 and13 add13 new13 users13 and13 programs13 to13 make13 it13 easier13 to13 access13 the13 virtual13 server13
To13 access13 the13 server13 go13 to13 the13 UNIX13 or13 Linux13 window13 that13 you13 used13 to13 create13 the13 SSH13 key13 In13 the13 directory13 where13 the13 ltuniquekeynamegt13 files13 exist13 run13 the13 following13 command13 referencing13 the13 SSH13 keyname13 the13 public13 IP13 address13 Bluemix13 allocated13 for13 the13 virtual13 server13 and13 the13 password13 for13 the13 SSH13 key13
[userserver ~]$ ssh -i ltuniquekeynamegt ibmcloudltpublicIPaddressgt
Enter13 the13 passphrase13 for13 the13 key13 ltuniquekeynamegt13
Last login Tue Jul 5 125546 2016 from 108-210-96-231lightspeedrlghncsbcglobalnet
[ibmcloudelkforcase ~]$
ibmcloud13 is13 the13 default13 user13 for13 the13 Bluemix13 virtual13 servers13 and13 allows13 SSH13 to13 access13 the13 server13
Acces13 the13 root13 user13 To13 access13 the13 root13 user13 log13 on13 as13 ibmcloud13 user13 and13 run13 the13 following13 command13
[ibmcloudelkforcase ~]$ sudo bash
[rootelkforcase ibmcloud]
The13 root13 user13 can13 be13 used13 to13 add13 programs13 to13 the13 server13 and13 create13 and13 edit13 users13 as13 required13 to13 access13 applicaons13 installed13 on13 the13 virtual13 server13
Page of 12 71 Page of 12 71
Copy13 files13 to13 the13 virtual13 server13 To13 move13 installaon13 files13 to13 the13 server13 go13 to13 the13 UNIX13 or13 Linux13 window13 you13 used13 to13 create13 the13 SSH13 key13 and13 the13 directory13 where13 the13 ltuniquekeynamegt13 files13 exist13 Move13 the13 installaon13 files13 to13 the13 same13 locaon13
Run13 the13 following13 command13 referencing13 the13 SSH13 keyname13 the13 public13 IP13 address13 Bluemix13 allocated13 for13 the13 virtual13 server13 and13 the13 password13 for13 the13 SSH13 key13
[userserver ~]$ scp ndashi ltuniquekeynamegt ltINSTALLATIONFILENAMEgt ibmcloudltpublicIPaddressgthomeibmcloud
Enter13 the13 passphrase13 for13 key13 cloudkey313
This13 copies13 the13 file13 using13 the13 ibmcloud13 user13 to13 the13 homeibmcloud13 directory13 on13 the13 virtual13 server13
Install13 and13 access13 the13 virtual13 server13 via13 VNC13 server13 Using13 the13 ssh13 command13 to13 access13 the13 virtual13 server13 is13 not13 the13 most13 stable13 method13 and13 does13 not13 offer13 X-shy‐Windows13 support13 Therefore13 we13 suggest13 that13 you13 use13 a13 VNC13 client13 to13 access13 the13 virtual13 server13
Installing13 a13 VNC13 client13 should13 be13 done13 as13 the13 root13 user13 Use13 the13 yum13 command13 as13 follows13
[ibmcloudelkforcase ~]$ yum groupinstall GNOME Desktop
[ibmcloudelkforcase ~]$ yum install tigervnc-server
These13 commands13 first13 install13 a13 desktop13 style13 OS13 user13 interface13 and13 then13 a13 VNC13 client13 to13 remotely13 access13 the13 desktop13 style13 OS13
Then13 as13 either13 the13 root13 user13 or13 another13 user13 (example13 below13 uses13 elk)13 run13 the13 following13 command13 to13 create13 the13 VNC13 client13 connecon13
[elkelkforcase ~]$ vncserver
New elkforcase2 (elk) desktop is elkforcase2
Starting applications specified in homeelkvncxstartup
Log file is homeelkvncelkforcase2log
[elkelkforcase ~]$
Install13 a13 VNC13 client13 on13 the13 client13 machine13 used13 to13 access13 the13 Internet13 and13 (in13 the13 example13 above)13 use13 the13 command13 ltpublicIPadddressgt213 and13 the13 password13 for13 ELK13 to13 access13 the13 virtual13 server13
Using13 a13 VNC13 client13 means13 that13 you13 can13 leave13 long13 duraon13 installaons13 running13 without13 fear13 of13 the13 terminal13 losing13 connecon13 and13 the13 installaons13 failing13
Install13 the13 ELK13 stack13 Download13 the13 source13 soaware13 code13 for13 the13 ELK13 stack13 from13 the13 ELK13 website13 13
At13 the13 me13 of13 wring13 the13 downloaded13 soaware13 files13 were13
Page of 13 71 Page of 13 71
13
Transfer13 the13 files13 to13 the13 virtual13 server13 (using13 the13 scp13 method13 detailed13 above)13
Install13 Elas0csearch13 As13 a13 root13 user13 create13 a13 non-shy‐root13 user13 with13 the13 adduser13 command13
Create13 a13 directory13 to13 install13 Elascsearch13 into13 and13 untar13 the13 contents13 of13 the13 Elascsearch13 install13 file13 into13 that13 directory13
[elkelkforcase]$ cd home
[elkelkforcase]$ mkdir elasticsearch
[elkelkforcase]$ cp homeibmcloudelasticsearch-233targz homeelasticsearch
[elkelkforcase]$ cd elasticsearch
[elkelkforcase]$ gunzip elasticsearch-233targz
[elkelkforcase]$ untar elasticsearch-233tar
Elascsearch13 is13 now13 installed13
Start13 Elas0csearch13 To13 start13 Elascsearch13 go13 to13 the13 bin13 directory13 as13 the13 non-shy‐root13 user13 when13 you13 installed13 it13 and13 run13 the13 elascsearch13 executable13
[elkelkforcase]$ cd homeelasticsearchelasticsearch-233bin
[elkelkforcase]$ elasticsearch amp
[2] 22211
[elkelkforcase]$ [2016-07-05 133857730][INFO ][node] [Arsenal] version[233] pid[22211] build[218bdf12016-05-17T154004Z]
[2016-07-05 133857740][INFO ][node] [Arsenal] initializing
[2016-07-05 133858984][INFO ][plugins] [Arsenal] modules [reindex lang-expression lang-groovy] plugins [] sites []
[2016-07-05 133859146][INFO ][env] [Arsenal] using [1] data paths mounts [[ (rootfs)]] net usable_space [155gb] net total_space [199gb] spins [unknown] types [rootfs]
[2016-07-05 133859146][INFO ][env] [Arsenal] heap size [10156mb] compressed ordinary object pointers [true]
Page of 14 71 Page of 14 71
[2016-07-05 133859147][WARN ][env] [Arsenal] max file descriptors [4096] for elasticsearch process likely too low consider increasing to at least [65536]
[2016-07-05 133903526][INFO ][node] [Arsenal] initialized
[2016-07-05 133903526][INFO ][node] [Arsenal] starting
To13 test13 that13 Elascsearch13 is13 running13 correctly13 open13 the13 browser13 in13 the13 VNC13 client13 and13 run13 localhost920013
13
Install13 Kibana13 ELK13 recommends13 installing13 and13 running13 Kibana13 as13 root13 13
1 As13 root13 user13 create13 an13 installaon13 directory13 and13 untar13 the13 installaon13 file13
[rootelkforcase]$ cd home
[rootelkforcase]$ mkdir kibana
[rootelkforcase]$ cp homeibmcloudkibana-451-linux-x64targz homekibana
[rootelkforcase]$ cd kibana
[rootelkforcase]$ gunzip kibana-451-linux-x64targz
[rootelkforcase]$ untar kibana-451-linux-x64tar
2 Modify13 the13 manifestyml13 configuraon13 file13 Go13 to13 the13 kibana-shy‐451-shy‐linux-shy‐x6413 directory13
[rootelkforcase]$ cd homekibanakibana-451-linux-x64
3 Edit13 the13 manifestyml13 file13 so13 the13 contents13 are13
applicaons13
-shy‐ name13 kibana-shy‐in-shy‐bluemix13
memory13 128M13
host13 kibana-shy‐in-shy‐bluemix13
buildpack13 hcpsgithubcomcloudfoundry-shy‐communitynginx-shy‐buildpackgit13
Page of 15 71 Page of 15 71
4 Edit13 the13 configkibanayml13 file13 so13 the13 contents13 are13
Kibana is served by a back end server This controls which port to use
port 5601
The host to bind the server to
host 0000
If you are running kibana behind a proxy and want to mount it at a path
specify that path here The basePath cant end in a slash
serverbasePath
The maximum payload size in bytes on incoming server requests
servermaxPayloadBytes 1048576
The Elasticsearch instance to use for all your queries
elasticsearchurl httplocalhost9200
preserve_elasticsearch_host true will send the hostname specified in `elasticsearch` If you set it to false
then the host you use to connect to this Kibana instance will be sent
elasticsearchpreserveHost true
Kibana uses an index in Elasticsearch to store saved searches visualizations
and dashboards It will create a new index if it doesnt already exist
kibanaindex kibana
The default application to load
kibanadefaultAppId discover
If your Elasticsearch is protected with basic auth these are the user credentials
used by the Kibana server to perform maintenance on the kibana_index at startup Your
Kibana users will still need to authenticate with Elasticsearch (which is proxied through
the Kibana server)
Page of 16 71 Page of 16 71
elasticsearchusername user
elasticsearchpassword pass
SSL for outgoing requests from the Kibana Server to the browser (PEM formatted)
serversslcert pathtoyourservercrt
serversslkey pathtoyourserverkey
Optional setting to validate that your Elasticsearch backend uses the same key files (PEM formatted)
elasticsearchsslcert pathtoyourclientcrt
elasticsearchsslkey pathtoyourclientkey
If you need to provide a CA certificate for your Elasticsearch instance put
the path of the pem file here
elasticsearchsslca pathtoyourCApem
Set to false to have a complete disregard for the validity of the SSL
certificate
elasticsearchsslverify true
Time in milliseconds to wait for elasticsearch to respond to pings defaults to
request_timeout setting
elasticsearchpingTimeout 1500
Time in milliseconds to wait for responses from the back end or elasticsearch
This must be gt 0
elasticsearchrequestTimeout 30000
Time in milliseconds for Elasticsearch to wait for responses from shards
Set to 0 to disable
elasticsearchshardTimeout 0
Time in milliseconds to wait for Elasticsearch at Kibana startup before retrying
elasticsearchstartupTimeout 5000
elasticsearchstartupTimeout 30000
Page of 17 71 Page of 17 71
Set the path to where you would like the process id file to be created
pidfile varrunkibanapid
If you would like to send the log output to a file you can set the path below
loggingdest stdout
Set this to true to suppress all logging output
loggingsilent false
Set this to true to suppress all logging output except for error messages
loggingquiet false
Set this to true to log all events including system usage information and all requests
loggingverbose false
Kibana13 is13 now13 installed13
Start13 Kibana13 To13 start13 Kibana13 go13 to13 the13 bin13 directory13 and13 run13 the13 kibana13 executable13
[rootelkforcase]$
[rootelkforcase]$ cd homekibanakibana-451-linux-x64bin
[rootelkforcase]$ kibana amp
[1] 22283
[rootelkforcase] log [095900542] [info][status][pluginkibana] Status changed from uninitialized to green - Ready
log [095900600] [info][status][pluginelasticsearch] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [095900689] [info][status][pluginkbn_vislib_vis_types] Status changed from uninitialized to green - Ready
log [095900717] [info][status][pluginmarkdown_vis] Status changed from uninitialized to green - Ready
log [095900721] [info][status][pluginmetric_vis] Status changed from uninitialized to green - Ready
log [095900814] [info][status][pluginspyModes] Status changed from uninitialized to green - Ready
log [095900934] [info][status][pluginstatusPage] Status changed from uninitialized to green - Ready
Page of 18 71 Page of 18 71
log [095900941] [info][status][plugintable_vis] Status changed from uninitialized to green - Ready
log [095901067] [info][listening] Server running at http00005601
log [095906083] [info][status][pluginelasticsearch] Status changed from yellow to yellow - No existing Kibana index found
log [095908942] [info][status][pluginelasticsearch] Status changed from yellow to green - Kibana index ready
Kibana13 is13 now13 started13
Broadcast13 Bluemix13 app13 log13 records13 Bluemix13 apps13 (which13 are13 shown13 in13 the13 US13 South13 Bluemix13 region13 of13 the13 System13 Architecture13 diagram)13 will13 produce13 logs13 in13 a13 syslog13 rfc542413 format13 using13 a13 Bluemix13 tool13 called13 Loggregator13
For13 Logstash13 to13 receive13 these13 logs13 each13 Bluemix13 app13 needs13 to13 broadcast13 these13 logs13 To13 do13 that13 run13 the13 following13 commands13 on13 each13 Bluemix13 app13 using13 the13 public13 IP13 address13 of13 the13 ELK13 virtual13 server13 the13 ELK13 tag13 to13 trace13 the13 process13 and13 insert13 the13 name13 of13 each13 app13 in13 turn13
cf cups elk -l syslogltpublicIPaddress5000 cf bind-service ltmyappgt elk
cf restage ltmyappgt
Install13 Logstash13 To13 install13 Logstash13 as13 root13 user13 create13 a13 directory13 for13 the13 installaon13 and13 untar13 the13 installaon13 file13 there13
13 13 [rootelkforcase]$ cd home
[rootelkforcase]$ mkdir logstash
[rootelkforcase]$ cp homeibmcloudlogstash-233targz homelogstash
[rootelkforcase]$ cd logstash
[rootelkforcase]$ gunzip logstash-233targz
[rootelkforcase]$ untar logstash-233tar
Write13 Logstash13 configura0on13 Logstash13 requires13 a13 configuraon13 file13 to13 be13 wricen13 to13 receive13 transform13 and13 output13 the13 log13 records13 to13 Elascsearch13 This13 secon13 explains13 the13 basic13 requirements13 of13 processing13 Bluemix13 app13 log13 records13 into13 Elascsearch13 (more13 complex13 configuraons13 are13 detailed13 later13 in13 this13 document13 for13 the13 integraons13 with13 third-shy‐party13 tools)13
Create13 a13 conf13 file13 as13 root13 user13
[rootelkforcase]$ cd homelogstashlogstash-233
[rootelkforcase]$ mkdir config
Page of 19 71 Page of 19 71
[rootelkforcase]$ cd config
[rootelkforcase]$ touch loggregatorconf
Edit13 the13 loggregator13 file13 with13 the13 following13 contents13
13 Input13 secon13 defines13 which13 log13 records13 to13 receive13
13 port13 and13 type13 reflect13 method13 selected13 for13 Cloud13 Foundry13 to13 broadcast13 the13 records13
input
tcp
port =gt 5000
type =gt syslog
13 Filter13 secon13 ndash13 transforms13 the13 data13
filter
Add a field to identify the records as Cloud Foundry based
mutate
add_field =gt format =gt cf
13 Output13 secon13 ndash13 sends13 the13 data13 to13 its13 desnaon(s)13
output
13 Use13 the13 elascsearch13 plugin13 to13 send13 the13 data13 to13 the13 elascsearch13 instance13
13 elasticsearch
hosts =gt localhost
stdout codec =gt rubydebug
13 End13 of13 Config13
Save13 the13 file13 Verify13 the13 format13 of13 the13 file13 is13 acceptable13 by13 running13 the13 following13 commands13
[rootelkforcase]$ cd homelogstashlogstash-233
[rootelkforcase]$ binlogstash agent --configtest -f configloggregatorconf
Page of 20 71 Page of 20 71
Configuraon13 is13 OK13
Start13 Logstash13 Aaer13 you13 have13 wricen13 and13 verified13 the13 Logstash13 configuraon13 you13 can13 start13 Logstash13 to13 process13 records13 from13 the13 Bluemix13 apps13 (via13 Cloud13 Foundry)13 Run13 the13 following13 as13 root13 user13
[rootelkforcase]$ cd homelogstashlogstash-233
[rootelkforcase]$ binlogstash agent -f configloggregatorconf
Logstash13 is13 now13 started13
Monitor13 Logstash13 processing13 Once13 Logstash13 is13 running13 and13 log13 records13 are13 being13 produced13 by13 the13 Bluemix13 apps13 you13 can13 monitor13 progress13 in13 the13 same13 window13 that13 was13 used13 to13 start13 Logstash13 Output13 is13 similar13 to13 the13 following13
13
The13 log13 records13 being13 broadcast13 from13 the13 Bluemix13 apps13 are13 now13 sent13 to13 Elascsearch13 from13 the13 Logstash13 output13 configuraon13 Records13 are13 sent13 in13 real13 me13 If13 Logstash13 is13 not13 running13 while13 log13 records13 are13 being13 produced13 then13 these13 records13 will13 not13 be13 processed13 by13 Logstash13 and13 will13 not13 be13 stored13 in13 Elascsearch13
Page of 21 71 Page of 21 71
Use13 ELK13 The13 log13 records13 from13 the13 Bluemix13 apps13 are13 now13 being13 stored13 and13 indexed13 by13 the13 Elascsearch13 component13 This13 data13 can13 be13 accessed13 via13 the13 Kibana13 applicaon13
Kibana13 Discover13 page13 Using13 a13 web13 browser13 either13 directly13 from13 a13 client13 or13 using13 the13 VNC13 client13 access13 Kibana13 using13 the13 publicIPaddress13 of13 the13 ELK13 virtual13 server13
hcpltpublicIPaddressgt560113
No13 userpassword13 credenals13 are13 required13
Kibana13 defaults13 to13 a13 search13 of13 all13 records13 ()13 in13 the13 last13 1513 minutes13 and13 returns13 the13 results13
13
More13 complex13 searches13 are13 possible13 by13 changing13 the13 me13 filter13 and13 supplying13 search13 queries13
Change13 the13 0me13 filter13 To13 change13 the13 me13 ranges13 click13 the13 clock13 symbol13 in13 the13 top-shy‐right13 of13 the13 Kibana13 Discover13 page13
13
You13 will13 see13 the13 following13 opons13
13
Page of 22 71 Page of 22 71
A13 variety13 of13 quick13 me13 ranges13 can13 be13 selected13 or13 you13 can13 switch13 to13 relave13 or13 absolute13 me13 ranges13
Quick13 me13 ranges13 allow13 switching13 between13 last13 X13 minutes13 to13 last13 Y13 hours13 or13 Z13 days13 etc13
Relave13 me13 ranges13 allow13 more13 specific13 versions13 of13 the13 Quick13 opons13 from13 specific13 minuteshoursdays13 to13 now13
13
Absolute13 me13 ranges13 specify13 the13 start13 and13 end13 mes13 for13 the13 searches13
13
Custom13 search13 queries13 You13 can13 use13 the13 text13 window13 of13 the13 Discover13 page13 to13 search13 for13 matches13 within13 the13 log13 records13 You13 can13 use13 Apache13 Lucene13 or13 Elascsearch13 query13 DSL13 syntax13
13
Page of 23 71 Page of 23 71
Saved13 searches13 You13 can13 save13 any13 custom13 search13 syntax13 and13 choice13 of13 me13 range13 for13 future13 ease13 of13 use13 Any13 relave13 or13 absolute13 me13 frame13 is13 saved13 as13 part13 of13 the13 search13 13
To13 save13 a13 search13
1 Set13 up13 and13 run13 the13 required13 syntax13 and13 me13 frame13
13
2 Click13 the13 Save13 Search13 icon13
13
3 Type13 in13 a13 name13 for13 the13 saved13 search13
Page of 24 71 Page of 24 71
13
7 To13 delete13 or13 update13 saved13 searches13 click13 13 manage13 saved13 searches13
13
Use13 the13 edit13 icons13 to13 make13 any13 changes13
Page of 26 71 Page of 26 71
13
Build13 simple13 Kibana13 histograms13 Kibana13 allows13 you13 to13 build13 charts13 and13 dashboards13 based13 on13 the13 search13 capability13 offered13 by13 the13 Discover13 page13 Saved13 searches13 can13 be13 used13 as13 the13 basis13 for13 charts13 and13 dashboards13 built13 using13 those13 charts13 To13 build13 a13 simple13 histogram13
1 Select13 the13 Visualize13 page13
13
2 Select13 Ver0cal13 Bar13 Chart13
13
3 To13 use13 a13 saved13 search13 click13 From13 a13 saved13 search13
Page of 27 71 Page of 27 71
13
4 Select13 a13 previously13 saved13 search13
13
5 Select13 to13 edit13 the13 x-shy‐axis
6 Edit13 the13 fields13 with13 informaon13 about13 the13 aggregaon13 field13 interval13 and13 custom13 label13
Page of 28 71 Page of 28 71
13
Page of 29 71 Page of 29 71
7 Click13 the13 green13 arrow13 The13 histogram13 is13 created13 and13 run13 based13 on13 the13 saved13 search
13
8 To13 save13 the13 histogram13 click13 the13 Save13 Visualiza0on13 icon13 and13 add13 a13 tle13
13
13
9 Click13 Save13
10 To13 view13 the13 visualisaon13 click13 the13 Dashboard13 page13
Page of 30 71 Page of 30 71
13
11 Click13 the13 Add13 Visualiza0on13 icon13
13
12 Select13 the13 visiualizaon13 that13 you13 just13 created13
13
Page of 31 71 Page of 31 71
13
13 To13 save13 the13 dashboard13 click13 the13 Save13 Dashboard13 icon13
Page of 32 71 Page of 32 71
13
14 Add13 a13 tle13 and13 select13 whether13 to13 store13 the13 selected13 me13 frame13 with13 the13 dashboard13
13
15 Click13 Save13
16 You13 can13 edit13 the13 shape13 and13 number13 of13 charts13 and13 save13 the13 dashboard13
17 To13 re-shy‐run13 the13 dashboard13 select13 the13 Dashboard13 page13 and13 click13 the13 Load13 Saved13 Dashboard13 icon13
Page of 33 71 Page of 33 71
13
18 Select13 the13 saved13 dashboard13 from13 the13 list13
13
19 The13 dashboard13 runs13
13
Page of 34 71 Page of 34 71
Page of 35 71 Page of 35 71
Aler0ng13 with13 ELK13 ELK13 examines13 the13 log13 records13 in13 process13 to13 send13 alerts13 to13 third-shy‐party13 alerng13 event13 management13 and13 social13 media13 applicaons13 As13 shown13 previously13 in13 the13 ELK13 architecture13 the13 ELK13 strategy13 for13 alerng13 is13 to13 send13 alerts13 from13 the13 Logstash13 component13 directly13 Therefore13 log13 records13 and13 alerts13 are13 sent13 to13 both13 Elascsearch13 and13 third-shy‐party13 tools13 at13 the13 same13 instant13 using13 the13 output13 secon13 of13 the13 Logstash13 configuraon13 Any13 alerts13 are13 generated13 during13 the13 filter13 secon13 processing13 of13 the13 Logstash13 configuraon13
Create13 alerts13 in13 Logstash13 The13 Logstash13 filter13 secon13 is13 used13 to13 trigger13 alerts13 based13 on13 the13 log13 records13 being13 processed13 For13 example13 log13 records13 can13 be13 checked13 for13 errors13 warnings13 crashed13 or13 stopped13 messages13 and13 generate13 an13 alert13
Here13 is13 an13 example13 Logstash13 filter13 code13
13 if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
The13 specific13 tags13 are13 set13 against13 the13 log13 records13 to13 trigger13 the13 alert13 later13 in13 the13 configuraon13 in13 the13 output13 secon13
Control13 the13 number13 of13 generated13 alerts13 using13 the13 Logstash13 throcle13 plugin13 so13 that13 either13 a13 lot13 of13 instances13 in13 a13 fixed13 me13 period13 are13 required13 before13 an13 alert13 is13 sent13 or13 a13 specific13 number13 of13 instances13 triggers13 the13 alert13
Page of 36 71 Page of 36 71
Edit13 the13 plugin13 parameters13 to13 meet13 system13 requirements13
13 13 13 13 13 if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 10
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle13
after_count =gt 20
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 30
period =gt 1200
key =gt servicename_alarm_warning
Page of 37 71 Page of 37 71
add_tag =gt servicename_throttled
13 13 13 13 13 13 13 13 13
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 40
period =gt 1500
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
Enrich13 alerts13 with13 message13 detail13 Some13 of13 the13 third-shy‐party13 integraons13 benefit13 from13 having13 components13 of13 the13 raw13 log13 record13 sent13 in13 addion13 to13 the13 raw13 record13 or13 fields13 in13 the13 record13 edited13 transformed13 or13 translated13
Grok13 Logstash13 allows13 the13 log13 record13 to13 be13 broken13 up13 into13 logical13 secons13 and13 each13 secon13 stored13 in13 a13 new13 parameter13 using13 the13 grok13 parameter13 in13 the13 filter13 secon13 Here13 is13 an13 example13 grok13
13 grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
To13 build13 the13 grok13 statement13 Logstash13 offers13 an13 online13 tool13 at13 hcpsgrokdebugherokuappcom13
Translate13 If13 specific13 fields13 in13 the13 log13 record13 are13 coded13 (for13 example13 the13 Bluemix13 app13 name13 is13 oaen13 not13 sent13 in13 the13 syslog13 messages13 instead13 the13 GUID13 is13 sent13 which13 is13 not13 useful)13 then13 the13 filter13 secon13 translate13 plugin13 can13 be13 used13 to13 add13 new13 fields13 by13 holding13 the13 decoded13 versions13 of13 those13 values13
13 translate
field =gt data1
destination =gt data1
override =gt true
Page of 38 71 Page of 38 71
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
dc50f6e4-c052-436a-bf27-95ba392c4228 monverifyservicemanagement
0c677e82-5bfc-4e38-9155-e416249e3ee6 ms-catalog-postpyretic-pomiculture
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
However13 the13 translate13 plugin13 is13 not13 included13 with13 the13 Logstash13 installaon13 In13 the13 next13 secon13 I13 will13 13 describe13 how13 to13 add13 new13 plugins13 to13 the13 Logstash13 instance13
Add13 custom13 plugins13 to13 Logstash13 As13 an13 open13 source13 tool13 Logstash13 encourages13 users13 to13 develop13 content13 for13 all13 users13 to13 share13 Some13 of13 the13 plugins13 that13 were13 created13 by13 our13 community13 include13 a13 translaon13 plugin13 an13 output13 Slack13 plugin13 and13 an13 output13 syslog13 plugin13 Letrsquos13 look13 at13 how13 to13 add13 these13 plugins13 to13 your13 environment13
Filter13 translate13 plugin13 To13 add13 the13 translate13 plugin13
1 Download13 the13 plugin13
2 Copy13 the13 plugin13 file13 logstash-shy‐filter-shy‐translate-shy‐214gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-filter-translate 214
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Output13 Slack13 plugin13 To13 add13 the13 Slack13 plugin13
Page of 39 71 Page of 39 71
1 Download13 the13 plugin13 13
2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐slack-shy‐014gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-output-slack 014
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Output13 syslog13 plugin13 To13 add13 the13 syslog13 plugin13
1 Download13 the13 plugin13 13
2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐syslog-shy‐214gem13 to13 the13 ELK13 Virtual13 Server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-output-syslog 214
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Integrate13 Logstash13 with13 Slack13 To13 send13 generated13 alerts13 to13 Slack13 complete13 these13 steps13
1 Create13 a13 Slack13 integraon13 13
2 Use13 the13 Slack13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 Slack13 integraon13
113 Login13 to13 Slack13 and13 go13 to13 the13 Apps13 page13 13
Page of 40 71 Page of 40 71
13
313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13
13
A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13
13
Page of 42 71 Page of 42 71
The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13
13
The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13
13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
slack
url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5
channel =gt cloud-operations-mon
username =gt nick_cawood
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13
Page of 43 71 Page of 43 71
Page of 44 71 Page of 44 71
In13 Slack13 the13 alerts13 look13 like13 this13
13
Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13
1 Create13 a13 PagerDuty13 integraon13 13
2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13
1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13
13
2 Click13 Add13 New13 Service13
13
313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13
Page of 45 71 Page of 45 71
13
413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13
513 Click13 Add13 Service13
13
The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13
Page of 46 71 Page of 46 71
13
The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt be7eb1ea9c224018923e818d3e891d0b
incident_key =gt logstashhosttype
Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13
Page of 47 71 Page of 47 71
13
Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13
1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13
2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13
3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13
4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13
13 13 http
url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo
http_method =gt post
format =gt json
For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13
13 13 syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
Page of 48 71 Page of 48 71
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13
13
The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13
if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2
This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13
Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13
Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13
Logstash13 severity Shown13 in13 Omnibus13 as
alert Indeterminate
crical Warning
debug 7
emergency Clear
error Minor
informaonal 6
noce Crical
warning Major
Page of 49 71 Page of 49 71
13
Page of 50 71 Page of 50 71
Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13
input
tcp
port =gt 5000
type =gt syslog
filter
mutate
add_field =gt format =gt cf
grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
translate
field =gt data1
destination =gt data1
override =gt true
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
=== Check Alarm Criteria ===
Standard Key Words
if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
=== Alarm Exclusions ===
if [message] =~ (i)sometexthere
mutate remove_tag =gt servicename_alarm
Throttle alarms so we do not hit PD rate limits
Page of 52 71 Page of 52 71
if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 5
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle
after_count =gt 5
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 5
period =gt 1200
key =gt servicename_alarm_warning
add_tag =gt servicename_throttled
Page of 53 71 Page of 53 71
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
output
elasticsearch
hosts =gt localhost
stdout codec =gt rubydebug
if servicename_alarm_err in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
Page of 54 71 Page of 54 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 55 71 Page of 55 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
Page of 56 71 Page of 56 71
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert err message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_error in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
Page of 57 71 Page of 57 71
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 58 71 Page of 58 71
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 59 71 Page of 59 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert error message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
Page of 60 71 Page of 60 71
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
Page of 61 71 Page of 61 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 62 71 Page of 62 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt x
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert fatal message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
Page of 63 71 Page of 63 71
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
Page of 64 71 Page of 64 71
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 65 71 Page of 65 71
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt heavy_exclamation_mark
Page of 66 71 Page of 66 71
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert WARNING message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 67 71 Page of 67 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
Page of 68 71 Page of 68 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 69 71 Page of 69 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
Page of 70 71 Page of 70 71
Page of 71 71 Page of 71 71
Provisioning13 a13 virtual13 server13 1 A13 Bluemix13 account13 is13 required13 for13 this13 process13 Sign13 up13 here13 and13 log13 in13 Make13 sure13 your13
environment13 is13 correct13 For13 example13 if13 you13 are13 in13 the13 United13 Kingdom13 select13 the13 United13 Kingdom13 region13 in13 the13 Bluemix13 console13
13
13
2 Add13 a13 virtual13 server13 to13 your13 Bluemix13 environment13 Click13 Run13 Virtual13 Servers13
13
3 Select13 an13 OS13 image13 from13 the13 EXISTING13 list13 For13 ELK13 select13 Centos13 OS13 (use13 latest13 available13 version)13
Page of 9 71 Page of 9 71
4 Name13 the13 virtual13 server13
5 Make13 sure13 Assign13 public13 IP13 address13 is13 checked13
13
6 Click13 Add13 Key13 to13 add13 the13 SSH13 key13 you13 created13 previously13 Copy13 the13 contents13 of13 the13 public13 key13 into13 the13 Public13 Key13 to13 import13 field13
Page of 10 71 Page of 10 71
13
7 Click13 OK13 Then13 click13 CREATE13
The13 virtual13 server13 is13 provisioned13 and13 added13 to13 the13 Bluemix13 dashboard13 under13 VIRTUAL13 SERVERS13
Page of 11 71 Page of 11 71
13
Set13 up13 the13 virtual13 server13
Log13 on13 to13 the13 virtual13 server13 Now13 you13 can13 log13 on13 to13 the13 virtual13 server13 and13 add13 new13 users13 and13 programs13 to13 make13 it13 easier13 to13 access13 the13 virtual13 server13
To13 access13 the13 server13 go13 to13 the13 UNIX13 or13 Linux13 window13 that13 you13 used13 to13 create13 the13 SSH13 key13 In13 the13 directory13 where13 the13 ltuniquekeynamegt13 files13 exist13 run13 the13 following13 command13 referencing13 the13 SSH13 keyname13 the13 public13 IP13 address13 Bluemix13 allocated13 for13 the13 virtual13 server13 and13 the13 password13 for13 the13 SSH13 key13
[userserver ~]$ ssh -i ltuniquekeynamegt ibmcloudltpublicIPaddressgt
Enter13 the13 passphrase13 for13 the13 key13 ltuniquekeynamegt13
Last login Tue Jul 5 125546 2016 from 108-210-96-231lightspeedrlghncsbcglobalnet
[ibmcloudelkforcase ~]$
ibmcloud13 is13 the13 default13 user13 for13 the13 Bluemix13 virtual13 servers13 and13 allows13 SSH13 to13 access13 the13 server13
Acces13 the13 root13 user13 To13 access13 the13 root13 user13 log13 on13 as13 ibmcloud13 user13 and13 run13 the13 following13 command13
[ibmcloudelkforcase ~]$ sudo bash
[rootelkforcase ibmcloud]
The13 root13 user13 can13 be13 used13 to13 add13 programs13 to13 the13 server13 and13 create13 and13 edit13 users13 as13 required13 to13 access13 applicaons13 installed13 on13 the13 virtual13 server13
Page of 12 71 Page of 12 71
Copy13 files13 to13 the13 virtual13 server13 To13 move13 installaon13 files13 to13 the13 server13 go13 to13 the13 UNIX13 or13 Linux13 window13 you13 used13 to13 create13 the13 SSH13 key13 and13 the13 directory13 where13 the13 ltuniquekeynamegt13 files13 exist13 Move13 the13 installaon13 files13 to13 the13 same13 locaon13
Run13 the13 following13 command13 referencing13 the13 SSH13 keyname13 the13 public13 IP13 address13 Bluemix13 allocated13 for13 the13 virtual13 server13 and13 the13 password13 for13 the13 SSH13 key13
[userserver ~]$ scp ndashi ltuniquekeynamegt ltINSTALLATIONFILENAMEgt ibmcloudltpublicIPaddressgthomeibmcloud
Enter13 the13 passphrase13 for13 key13 cloudkey313
This13 copies13 the13 file13 using13 the13 ibmcloud13 user13 to13 the13 homeibmcloud13 directory13 on13 the13 virtual13 server13
Install13 and13 access13 the13 virtual13 server13 via13 VNC13 server13 Using13 the13 ssh13 command13 to13 access13 the13 virtual13 server13 is13 not13 the13 most13 stable13 method13 and13 does13 not13 offer13 X-shy‐Windows13 support13 Therefore13 we13 suggest13 that13 you13 use13 a13 VNC13 client13 to13 access13 the13 virtual13 server13
Installing13 a13 VNC13 client13 should13 be13 done13 as13 the13 root13 user13 Use13 the13 yum13 command13 as13 follows13
[ibmcloudelkforcase ~]$ yum groupinstall GNOME Desktop
[ibmcloudelkforcase ~]$ yum install tigervnc-server
These13 commands13 first13 install13 a13 desktop13 style13 OS13 user13 interface13 and13 then13 a13 VNC13 client13 to13 remotely13 access13 the13 desktop13 style13 OS13
Then13 as13 either13 the13 root13 user13 or13 another13 user13 (example13 below13 uses13 elk)13 run13 the13 following13 command13 to13 create13 the13 VNC13 client13 connecon13
[elkelkforcase ~]$ vncserver
New elkforcase2 (elk) desktop is elkforcase2
Starting applications specified in homeelkvncxstartup
Log file is homeelkvncelkforcase2log
[elkelkforcase ~]$
Install13 a13 VNC13 client13 on13 the13 client13 machine13 used13 to13 access13 the13 Internet13 and13 (in13 the13 example13 above)13 use13 the13 command13 ltpublicIPadddressgt213 and13 the13 password13 for13 ELK13 to13 access13 the13 virtual13 server13
Using13 a13 VNC13 client13 means13 that13 you13 can13 leave13 long13 duraon13 installaons13 running13 without13 fear13 of13 the13 terminal13 losing13 connecon13 and13 the13 installaons13 failing13
Install13 the13 ELK13 stack13 Download13 the13 source13 soaware13 code13 for13 the13 ELK13 stack13 from13 the13 ELK13 website13 13
At13 the13 me13 of13 wring13 the13 downloaded13 soaware13 files13 were13
Page of 13 71 Page of 13 71
13
Transfer13 the13 files13 to13 the13 virtual13 server13 (using13 the13 scp13 method13 detailed13 above)13
Install13 Elas0csearch13 As13 a13 root13 user13 create13 a13 non-shy‐root13 user13 with13 the13 adduser13 command13
Create13 a13 directory13 to13 install13 Elascsearch13 into13 and13 untar13 the13 contents13 of13 the13 Elascsearch13 install13 file13 into13 that13 directory13
[elkelkforcase]$ cd home
[elkelkforcase]$ mkdir elasticsearch
[elkelkforcase]$ cp homeibmcloudelasticsearch-233targz homeelasticsearch
[elkelkforcase]$ cd elasticsearch
[elkelkforcase]$ gunzip elasticsearch-233targz
[elkelkforcase]$ untar elasticsearch-233tar
Elascsearch13 is13 now13 installed13
Start13 Elas0csearch13 To13 start13 Elascsearch13 go13 to13 the13 bin13 directory13 as13 the13 non-shy‐root13 user13 when13 you13 installed13 it13 and13 run13 the13 elascsearch13 executable13
[elkelkforcase]$ cd homeelasticsearchelasticsearch-233bin
[elkelkforcase]$ elasticsearch amp
[2] 22211
[elkelkforcase]$ [2016-07-05 133857730][INFO ][node] [Arsenal] version[233] pid[22211] build[218bdf12016-05-17T154004Z]
[2016-07-05 133857740][INFO ][node] [Arsenal] initializing
[2016-07-05 133858984][INFO ][plugins] [Arsenal] modules [reindex lang-expression lang-groovy] plugins [] sites []
[2016-07-05 133859146][INFO ][env] [Arsenal] using [1] data paths mounts [[ (rootfs)]] net usable_space [155gb] net total_space [199gb] spins [unknown] types [rootfs]
[2016-07-05 133859146][INFO ][env] [Arsenal] heap size [10156mb] compressed ordinary object pointers [true]
Page of 14 71 Page of 14 71
[2016-07-05 133859147][WARN ][env] [Arsenal] max file descriptors [4096] for elasticsearch process likely too low consider increasing to at least [65536]
[2016-07-05 133903526][INFO ][node] [Arsenal] initialized
[2016-07-05 133903526][INFO ][node] [Arsenal] starting
To13 test13 that13 Elascsearch13 is13 running13 correctly13 open13 the13 browser13 in13 the13 VNC13 client13 and13 run13 localhost920013
13
Install13 Kibana13 ELK13 recommends13 installing13 and13 running13 Kibana13 as13 root13 13
1 As13 root13 user13 create13 an13 installaon13 directory13 and13 untar13 the13 installaon13 file13
[rootelkforcase]$ cd home
[rootelkforcase]$ mkdir kibana
[rootelkforcase]$ cp homeibmcloudkibana-451-linux-x64targz homekibana
[rootelkforcase]$ cd kibana
[rootelkforcase]$ gunzip kibana-451-linux-x64targz
[rootelkforcase]$ untar kibana-451-linux-x64tar
2 Modify13 the13 manifestyml13 configuraon13 file13 Go13 to13 the13 kibana-shy‐451-shy‐linux-shy‐x6413 directory13
[rootelkforcase]$ cd homekibanakibana-451-linux-x64
3 Edit13 the13 manifestyml13 file13 so13 the13 contents13 are13
applicaons13
-shy‐ name13 kibana-shy‐in-shy‐bluemix13
memory13 128M13
host13 kibana-shy‐in-shy‐bluemix13
buildpack13 hcpsgithubcomcloudfoundry-shy‐communitynginx-shy‐buildpackgit13
Page of 15 71 Page of 15 71
4 Edit13 the13 configkibanayml13 file13 so13 the13 contents13 are13
Kibana is served by a back end server This controls which port to use
port 5601
The host to bind the server to
host 0000
If you are running kibana behind a proxy and want to mount it at a path
specify that path here The basePath cant end in a slash
serverbasePath
The maximum payload size in bytes on incoming server requests
servermaxPayloadBytes 1048576
The Elasticsearch instance to use for all your queries
elasticsearchurl httplocalhost9200
preserve_elasticsearch_host true will send the hostname specified in `elasticsearch` If you set it to false
then the host you use to connect to this Kibana instance will be sent
elasticsearchpreserveHost true
Kibana uses an index in Elasticsearch to store saved searches visualizations
and dashboards It will create a new index if it doesnt already exist
kibanaindex kibana
The default application to load
kibanadefaultAppId discover
If your Elasticsearch is protected with basic auth these are the user credentials
used by the Kibana server to perform maintenance on the kibana_index at startup Your
Kibana users will still need to authenticate with Elasticsearch (which is proxied through
the Kibana server)
Page of 16 71 Page of 16 71
elasticsearchusername user
elasticsearchpassword pass
SSL for outgoing requests from the Kibana Server to the browser (PEM formatted)
serversslcert pathtoyourservercrt
serversslkey pathtoyourserverkey
Optional setting to validate that your Elasticsearch backend uses the same key files (PEM formatted)
elasticsearchsslcert pathtoyourclientcrt
elasticsearchsslkey pathtoyourclientkey
If you need to provide a CA certificate for your Elasticsearch instance put
the path of the pem file here
elasticsearchsslca pathtoyourCApem
Set to false to have a complete disregard for the validity of the SSL
certificate
elasticsearchsslverify true
Time in milliseconds to wait for elasticsearch to respond to pings defaults to
request_timeout setting
elasticsearchpingTimeout 1500
Time in milliseconds to wait for responses from the back end or elasticsearch
This must be gt 0
elasticsearchrequestTimeout 30000
Time in milliseconds for Elasticsearch to wait for responses from shards
Set to 0 to disable
elasticsearchshardTimeout 0
Time in milliseconds to wait for Elasticsearch at Kibana startup before retrying
elasticsearchstartupTimeout 5000
elasticsearchstartupTimeout 30000
Page of 17 71 Page of 17 71
Set the path to where you would like the process id file to be created
pidfile varrunkibanapid
If you would like to send the log output to a file you can set the path below
loggingdest stdout
Set this to true to suppress all logging output
loggingsilent false
Set this to true to suppress all logging output except for error messages
loggingquiet false
Set this to true to log all events including system usage information and all requests
loggingverbose false
Kibana13 is13 now13 installed13
Start13 Kibana13 To13 start13 Kibana13 go13 to13 the13 bin13 directory13 and13 run13 the13 kibana13 executable13
[rootelkforcase]$
[rootelkforcase]$ cd homekibanakibana-451-linux-x64bin
[rootelkforcase]$ kibana amp
[1] 22283
[rootelkforcase] log [095900542] [info][status][pluginkibana] Status changed from uninitialized to green - Ready
log [095900600] [info][status][pluginelasticsearch] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [095900689] [info][status][pluginkbn_vislib_vis_types] Status changed from uninitialized to green - Ready
log [095900717] [info][status][pluginmarkdown_vis] Status changed from uninitialized to green - Ready
log [095900721] [info][status][pluginmetric_vis] Status changed from uninitialized to green - Ready
log [095900814] [info][status][pluginspyModes] Status changed from uninitialized to green - Ready
log [095900934] [info][status][pluginstatusPage] Status changed from uninitialized to green - Ready
Page of 18 71 Page of 18 71
log [095900941] [info][status][plugintable_vis] Status changed from uninitialized to green - Ready
log [095901067] [info][listening] Server running at http00005601
log [095906083] [info][status][pluginelasticsearch] Status changed from yellow to yellow - No existing Kibana index found
log [095908942] [info][status][pluginelasticsearch] Status changed from yellow to green - Kibana index ready
Kibana13 is13 now13 started13
Broadcast13 Bluemix13 app13 log13 records13 Bluemix13 apps13 (which13 are13 shown13 in13 the13 US13 South13 Bluemix13 region13 of13 the13 System13 Architecture13 diagram)13 will13 produce13 logs13 in13 a13 syslog13 rfc542413 format13 using13 a13 Bluemix13 tool13 called13 Loggregator13
For13 Logstash13 to13 receive13 these13 logs13 each13 Bluemix13 app13 needs13 to13 broadcast13 these13 logs13 To13 do13 that13 run13 the13 following13 commands13 on13 each13 Bluemix13 app13 using13 the13 public13 IP13 address13 of13 the13 ELK13 virtual13 server13 the13 ELK13 tag13 to13 trace13 the13 process13 and13 insert13 the13 name13 of13 each13 app13 in13 turn13
cf cups elk -l syslogltpublicIPaddress5000 cf bind-service ltmyappgt elk
cf restage ltmyappgt
Install13 Logstash13 To13 install13 Logstash13 as13 root13 user13 create13 a13 directory13 for13 the13 installaon13 and13 untar13 the13 installaon13 file13 there13
13 13 [rootelkforcase]$ cd home
[rootelkforcase]$ mkdir logstash
[rootelkforcase]$ cp homeibmcloudlogstash-233targz homelogstash
[rootelkforcase]$ cd logstash
[rootelkforcase]$ gunzip logstash-233targz
[rootelkforcase]$ untar logstash-233tar
Write13 Logstash13 configura0on13 Logstash13 requires13 a13 configuraon13 file13 to13 be13 wricen13 to13 receive13 transform13 and13 output13 the13 log13 records13 to13 Elascsearch13 This13 secon13 explains13 the13 basic13 requirements13 of13 processing13 Bluemix13 app13 log13 records13 into13 Elascsearch13 (more13 complex13 configuraons13 are13 detailed13 later13 in13 this13 document13 for13 the13 integraons13 with13 third-shy‐party13 tools)13
Create13 a13 conf13 file13 as13 root13 user13
[rootelkforcase]$ cd homelogstashlogstash-233
[rootelkforcase]$ mkdir config
Page of 19 71 Page of 19 71
[rootelkforcase]$ cd config
[rootelkforcase]$ touch loggregatorconf
Edit13 the13 loggregator13 file13 with13 the13 following13 contents13
13 Input13 secon13 defines13 which13 log13 records13 to13 receive13
13 port13 and13 type13 reflect13 method13 selected13 for13 Cloud13 Foundry13 to13 broadcast13 the13 records13
input
tcp
port =gt 5000
type =gt syslog
13 Filter13 secon13 ndash13 transforms13 the13 data13
filter
Add a field to identify the records as Cloud Foundry based
mutate
add_field =gt format =gt cf
13 Output13 secon13 ndash13 sends13 the13 data13 to13 its13 desnaon(s)13
output
13 Use13 the13 elascsearch13 plugin13 to13 send13 the13 data13 to13 the13 elascsearch13 instance13
13 elasticsearch
hosts =gt localhost
stdout codec =gt rubydebug
13 End13 of13 Config13
Save13 the13 file13 Verify13 the13 format13 of13 the13 file13 is13 acceptable13 by13 running13 the13 following13 commands13
[rootelkforcase]$ cd homelogstashlogstash-233
[rootelkforcase]$ binlogstash agent --configtest -f configloggregatorconf
Page of 20 71 Page of 20 71
Configuraon13 is13 OK13
Start13 Logstash13 Aaer13 you13 have13 wricen13 and13 verified13 the13 Logstash13 configuraon13 you13 can13 start13 Logstash13 to13 process13 records13 from13 the13 Bluemix13 apps13 (via13 Cloud13 Foundry)13 Run13 the13 following13 as13 root13 user13
[rootelkforcase]$ cd homelogstashlogstash-233
[rootelkforcase]$ binlogstash agent -f configloggregatorconf
Logstash13 is13 now13 started13
Monitor13 Logstash13 processing13 Once13 Logstash13 is13 running13 and13 log13 records13 are13 being13 produced13 by13 the13 Bluemix13 apps13 you13 can13 monitor13 progress13 in13 the13 same13 window13 that13 was13 used13 to13 start13 Logstash13 Output13 is13 similar13 to13 the13 following13
13
The13 log13 records13 being13 broadcast13 from13 the13 Bluemix13 apps13 are13 now13 sent13 to13 Elascsearch13 from13 the13 Logstash13 output13 configuraon13 Records13 are13 sent13 in13 real13 me13 If13 Logstash13 is13 not13 running13 while13 log13 records13 are13 being13 produced13 then13 these13 records13 will13 not13 be13 processed13 by13 Logstash13 and13 will13 not13 be13 stored13 in13 Elascsearch13
Page of 21 71 Page of 21 71
Use13 ELK13 The13 log13 records13 from13 the13 Bluemix13 apps13 are13 now13 being13 stored13 and13 indexed13 by13 the13 Elascsearch13 component13 This13 data13 can13 be13 accessed13 via13 the13 Kibana13 applicaon13
Kibana13 Discover13 page13 Using13 a13 web13 browser13 either13 directly13 from13 a13 client13 or13 using13 the13 VNC13 client13 access13 Kibana13 using13 the13 publicIPaddress13 of13 the13 ELK13 virtual13 server13
hcpltpublicIPaddressgt560113
No13 userpassword13 credenals13 are13 required13
Kibana13 defaults13 to13 a13 search13 of13 all13 records13 ()13 in13 the13 last13 1513 minutes13 and13 returns13 the13 results13
13
More13 complex13 searches13 are13 possible13 by13 changing13 the13 me13 filter13 and13 supplying13 search13 queries13
Change13 the13 0me13 filter13 To13 change13 the13 me13 ranges13 click13 the13 clock13 symbol13 in13 the13 top-shy‐right13 of13 the13 Kibana13 Discover13 page13
13
You13 will13 see13 the13 following13 opons13
13
Page of 22 71 Page of 22 71
A13 variety13 of13 quick13 me13 ranges13 can13 be13 selected13 or13 you13 can13 switch13 to13 relave13 or13 absolute13 me13 ranges13
Quick13 me13 ranges13 allow13 switching13 between13 last13 X13 minutes13 to13 last13 Y13 hours13 or13 Z13 days13 etc13
Relave13 me13 ranges13 allow13 more13 specific13 versions13 of13 the13 Quick13 opons13 from13 specific13 minuteshoursdays13 to13 now13
13
Absolute13 me13 ranges13 specify13 the13 start13 and13 end13 mes13 for13 the13 searches13
13
Custom13 search13 queries13 You13 can13 use13 the13 text13 window13 of13 the13 Discover13 page13 to13 search13 for13 matches13 within13 the13 log13 records13 You13 can13 use13 Apache13 Lucene13 or13 Elascsearch13 query13 DSL13 syntax13
13
Page of 23 71 Page of 23 71
Saved13 searches13 You13 can13 save13 any13 custom13 search13 syntax13 and13 choice13 of13 me13 range13 for13 future13 ease13 of13 use13 Any13 relave13 or13 absolute13 me13 frame13 is13 saved13 as13 part13 of13 the13 search13 13
To13 save13 a13 search13
1 Set13 up13 and13 run13 the13 required13 syntax13 and13 me13 frame13
13
2 Click13 the13 Save13 Search13 icon13
13
3 Type13 in13 a13 name13 for13 the13 saved13 search13
Page of 24 71 Page of 24 71
13
7 To13 delete13 or13 update13 saved13 searches13 click13 13 manage13 saved13 searches13
13
Use13 the13 edit13 icons13 to13 make13 any13 changes13
Page of 26 71 Page of 26 71
13
Build13 simple13 Kibana13 histograms13 Kibana13 allows13 you13 to13 build13 charts13 and13 dashboards13 based13 on13 the13 search13 capability13 offered13 by13 the13 Discover13 page13 Saved13 searches13 can13 be13 used13 as13 the13 basis13 for13 charts13 and13 dashboards13 built13 using13 those13 charts13 To13 build13 a13 simple13 histogram13
1 Select13 the13 Visualize13 page13
13
2 Select13 Ver0cal13 Bar13 Chart13
13
3 To13 use13 a13 saved13 search13 click13 From13 a13 saved13 search13
Page of 27 71 Page of 27 71
13
4 Select13 a13 previously13 saved13 search13
13
5 Select13 to13 edit13 the13 x-shy‐axis
6 Edit13 the13 fields13 with13 informaon13 about13 the13 aggregaon13 field13 interval13 and13 custom13 label13
Page of 28 71 Page of 28 71
13
Page of 29 71 Page of 29 71
7 Click13 the13 green13 arrow13 The13 histogram13 is13 created13 and13 run13 based13 on13 the13 saved13 search
13
8 To13 save13 the13 histogram13 click13 the13 Save13 Visualiza0on13 icon13 and13 add13 a13 tle13
13
13
9 Click13 Save13
10 To13 view13 the13 visualisaon13 click13 the13 Dashboard13 page13
Page of 30 71 Page of 30 71
13
11 Click13 the13 Add13 Visualiza0on13 icon13
13
12 Select13 the13 visiualizaon13 that13 you13 just13 created13
13
Page of 31 71 Page of 31 71
13
13 To13 save13 the13 dashboard13 click13 the13 Save13 Dashboard13 icon13
Page of 32 71 Page of 32 71
13
14 Add13 a13 tle13 and13 select13 whether13 to13 store13 the13 selected13 me13 frame13 with13 the13 dashboard13
13
15 Click13 Save13
16 You13 can13 edit13 the13 shape13 and13 number13 of13 charts13 and13 save13 the13 dashboard13
17 To13 re-shy‐run13 the13 dashboard13 select13 the13 Dashboard13 page13 and13 click13 the13 Load13 Saved13 Dashboard13 icon13
Page of 33 71 Page of 33 71
13
18 Select13 the13 saved13 dashboard13 from13 the13 list13
13
19 The13 dashboard13 runs13
13
Page of 34 71 Page of 34 71
Page of 35 71 Page of 35 71
Aler0ng13 with13 ELK13 ELK13 examines13 the13 log13 records13 in13 process13 to13 send13 alerts13 to13 third-shy‐party13 alerng13 event13 management13 and13 social13 media13 applicaons13 As13 shown13 previously13 in13 the13 ELK13 architecture13 the13 ELK13 strategy13 for13 alerng13 is13 to13 send13 alerts13 from13 the13 Logstash13 component13 directly13 Therefore13 log13 records13 and13 alerts13 are13 sent13 to13 both13 Elascsearch13 and13 third-shy‐party13 tools13 at13 the13 same13 instant13 using13 the13 output13 secon13 of13 the13 Logstash13 configuraon13 Any13 alerts13 are13 generated13 during13 the13 filter13 secon13 processing13 of13 the13 Logstash13 configuraon13
Create13 alerts13 in13 Logstash13 The13 Logstash13 filter13 secon13 is13 used13 to13 trigger13 alerts13 based13 on13 the13 log13 records13 being13 processed13 For13 example13 log13 records13 can13 be13 checked13 for13 errors13 warnings13 crashed13 or13 stopped13 messages13 and13 generate13 an13 alert13
Here13 is13 an13 example13 Logstash13 filter13 code13
13 if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
The13 specific13 tags13 are13 set13 against13 the13 log13 records13 to13 trigger13 the13 alert13 later13 in13 the13 configuraon13 in13 the13 output13 secon13
Control13 the13 number13 of13 generated13 alerts13 using13 the13 Logstash13 throcle13 plugin13 so13 that13 either13 a13 lot13 of13 instances13 in13 a13 fixed13 me13 period13 are13 required13 before13 an13 alert13 is13 sent13 or13 a13 specific13 number13 of13 instances13 triggers13 the13 alert13
Page of 36 71 Page of 36 71
Edit13 the13 plugin13 parameters13 to13 meet13 system13 requirements13
13 13 13 13 13 if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 10
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle13
after_count =gt 20
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 30
period =gt 1200
key =gt servicename_alarm_warning
Page of 37 71 Page of 37 71
add_tag =gt servicename_throttled
13 13 13 13 13 13 13 13 13
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 40
period =gt 1500
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
Enrich13 alerts13 with13 message13 detail13 Some13 of13 the13 third-shy‐party13 integraons13 benefit13 from13 having13 components13 of13 the13 raw13 log13 record13 sent13 in13 addion13 to13 the13 raw13 record13 or13 fields13 in13 the13 record13 edited13 transformed13 or13 translated13
Grok13 Logstash13 allows13 the13 log13 record13 to13 be13 broken13 up13 into13 logical13 secons13 and13 each13 secon13 stored13 in13 a13 new13 parameter13 using13 the13 grok13 parameter13 in13 the13 filter13 secon13 Here13 is13 an13 example13 grok13
13 grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
To13 build13 the13 grok13 statement13 Logstash13 offers13 an13 online13 tool13 at13 hcpsgrokdebugherokuappcom13
Translate13 If13 specific13 fields13 in13 the13 log13 record13 are13 coded13 (for13 example13 the13 Bluemix13 app13 name13 is13 oaen13 not13 sent13 in13 the13 syslog13 messages13 instead13 the13 GUID13 is13 sent13 which13 is13 not13 useful)13 then13 the13 filter13 secon13 translate13 plugin13 can13 be13 used13 to13 add13 new13 fields13 by13 holding13 the13 decoded13 versions13 of13 those13 values13
13 translate
field =gt data1
destination =gt data1
override =gt true
Page of 38 71 Page of 38 71
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
dc50f6e4-c052-436a-bf27-95ba392c4228 monverifyservicemanagement
0c677e82-5bfc-4e38-9155-e416249e3ee6 ms-catalog-postpyretic-pomiculture
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
However13 the13 translate13 plugin13 is13 not13 included13 with13 the13 Logstash13 installaon13 In13 the13 next13 secon13 I13 will13 13 describe13 how13 to13 add13 new13 plugins13 to13 the13 Logstash13 instance13
Add13 custom13 plugins13 to13 Logstash13 As13 an13 open13 source13 tool13 Logstash13 encourages13 users13 to13 develop13 content13 for13 all13 users13 to13 share13 Some13 of13 the13 plugins13 that13 were13 created13 by13 our13 community13 include13 a13 translaon13 plugin13 an13 output13 Slack13 plugin13 and13 an13 output13 syslog13 plugin13 Letrsquos13 look13 at13 how13 to13 add13 these13 plugins13 to13 your13 environment13
Filter13 translate13 plugin13 To13 add13 the13 translate13 plugin13
1 Download13 the13 plugin13
2 Copy13 the13 plugin13 file13 logstash-shy‐filter-shy‐translate-shy‐214gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-filter-translate 214
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Output13 Slack13 plugin13 To13 add13 the13 Slack13 plugin13
Page of 39 71 Page of 39 71
1 Download13 the13 plugin13 13
2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐slack-shy‐014gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-output-slack 014
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Output13 syslog13 plugin13 To13 add13 the13 syslog13 plugin13
1 Download13 the13 plugin13 13
2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐syslog-shy‐214gem13 to13 the13 ELK13 Virtual13 Server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-output-syslog 214
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Integrate13 Logstash13 with13 Slack13 To13 send13 generated13 alerts13 to13 Slack13 complete13 these13 steps13
1 Create13 a13 Slack13 integraon13 13
2 Use13 the13 Slack13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 Slack13 integraon13
113 Login13 to13 Slack13 and13 go13 to13 the13 Apps13 page13 13
Page of 40 71 Page of 40 71
13
313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13
13
A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13
13
Page of 42 71 Page of 42 71
The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13
13
The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13
13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
slack
url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5
channel =gt cloud-operations-mon
username =gt nick_cawood
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13
Page of 43 71 Page of 43 71
Page of 44 71 Page of 44 71
In13 Slack13 the13 alerts13 look13 like13 this13
13
Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13
1 Create13 a13 PagerDuty13 integraon13 13
2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13
1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13
13
2 Click13 Add13 New13 Service13
13
313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13
Page of 45 71 Page of 45 71
13
413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13
513 Click13 Add13 Service13
13
The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13
Page of 46 71 Page of 46 71
13
The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt be7eb1ea9c224018923e818d3e891d0b
incident_key =gt logstashhosttype
Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13
Page of 47 71 Page of 47 71
13
Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13
1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13
2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13
3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13
4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13
13 13 http
url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo
http_method =gt post
format =gt json
For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13
13 13 syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
Page of 48 71 Page of 48 71
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13
13
The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13
if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2
This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13
Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13
Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13
Logstash13 severity Shown13 in13 Omnibus13 as
alert Indeterminate
crical Warning
debug 7
emergency Clear
error Minor
informaonal 6
noce Crical
warning Major
Page of 49 71 Page of 49 71
13
Page of 50 71 Page of 50 71
Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13
input
tcp
port =gt 5000
type =gt syslog
filter
mutate
add_field =gt format =gt cf
grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
translate
field =gt data1
destination =gt data1
override =gt true
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
=== Check Alarm Criteria ===
Standard Key Words
if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
=== Alarm Exclusions ===
if [message] =~ (i)sometexthere
mutate remove_tag =gt servicename_alarm
Throttle alarms so we do not hit PD rate limits
Page of 52 71 Page of 52 71
if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 5
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle
after_count =gt 5
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 5
period =gt 1200
key =gt servicename_alarm_warning
add_tag =gt servicename_throttled
Page of 53 71 Page of 53 71
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
output
elasticsearch
hosts =gt localhost
stdout codec =gt rubydebug
if servicename_alarm_err in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
Page of 54 71 Page of 54 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 55 71 Page of 55 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
Page of 56 71 Page of 56 71
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert err message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_error in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
Page of 57 71 Page of 57 71
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 58 71 Page of 58 71
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 59 71 Page of 59 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert error message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
Page of 60 71 Page of 60 71
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
Page of 61 71 Page of 61 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 62 71 Page of 62 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt x
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert fatal message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
Page of 63 71 Page of 63 71
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
Page of 64 71 Page of 64 71
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 65 71 Page of 65 71
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt heavy_exclamation_mark
Page of 66 71 Page of 66 71
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert WARNING message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 67 71 Page of 67 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
Page of 68 71 Page of 68 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 69 71 Page of 69 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
Page of 70 71 Page of 70 71
Page of 71 71 Page of 71 71
4 Name13 the13 virtual13 server13
5 Make13 sure13 Assign13 public13 IP13 address13 is13 checked13
13
6 Click13 Add13 Key13 to13 add13 the13 SSH13 key13 you13 created13 previously13 Copy13 the13 contents13 of13 the13 public13 key13 into13 the13 Public13 Key13 to13 import13 field13
Page of 10 71 Page of 10 71
13
7 Click13 OK13 Then13 click13 CREATE13
The13 virtual13 server13 is13 provisioned13 and13 added13 to13 the13 Bluemix13 dashboard13 under13 VIRTUAL13 SERVERS13
Page of 11 71 Page of 11 71
13
Set13 up13 the13 virtual13 server13
Log13 on13 to13 the13 virtual13 server13 Now13 you13 can13 log13 on13 to13 the13 virtual13 server13 and13 add13 new13 users13 and13 programs13 to13 make13 it13 easier13 to13 access13 the13 virtual13 server13
To13 access13 the13 server13 go13 to13 the13 UNIX13 or13 Linux13 window13 that13 you13 used13 to13 create13 the13 SSH13 key13 In13 the13 directory13 where13 the13 ltuniquekeynamegt13 files13 exist13 run13 the13 following13 command13 referencing13 the13 SSH13 keyname13 the13 public13 IP13 address13 Bluemix13 allocated13 for13 the13 virtual13 server13 and13 the13 password13 for13 the13 SSH13 key13
[userserver ~]$ ssh -i ltuniquekeynamegt ibmcloudltpublicIPaddressgt
Enter13 the13 passphrase13 for13 the13 key13 ltuniquekeynamegt13
Last login Tue Jul 5 125546 2016 from 108-210-96-231lightspeedrlghncsbcglobalnet
[ibmcloudelkforcase ~]$
ibmcloud13 is13 the13 default13 user13 for13 the13 Bluemix13 virtual13 servers13 and13 allows13 SSH13 to13 access13 the13 server13
Acces13 the13 root13 user13 To13 access13 the13 root13 user13 log13 on13 as13 ibmcloud13 user13 and13 run13 the13 following13 command13
[ibmcloudelkforcase ~]$ sudo bash
[rootelkforcase ibmcloud]
The13 root13 user13 can13 be13 used13 to13 add13 programs13 to13 the13 server13 and13 create13 and13 edit13 users13 as13 required13 to13 access13 applicaons13 installed13 on13 the13 virtual13 server13
Page of 12 71 Page of 12 71
Copy13 files13 to13 the13 virtual13 server13 To13 move13 installaon13 files13 to13 the13 server13 go13 to13 the13 UNIX13 or13 Linux13 window13 you13 used13 to13 create13 the13 SSH13 key13 and13 the13 directory13 where13 the13 ltuniquekeynamegt13 files13 exist13 Move13 the13 installaon13 files13 to13 the13 same13 locaon13
Run13 the13 following13 command13 referencing13 the13 SSH13 keyname13 the13 public13 IP13 address13 Bluemix13 allocated13 for13 the13 virtual13 server13 and13 the13 password13 for13 the13 SSH13 key13
[userserver ~]$ scp ndashi ltuniquekeynamegt ltINSTALLATIONFILENAMEgt ibmcloudltpublicIPaddressgthomeibmcloud
Enter13 the13 passphrase13 for13 key13 cloudkey313
This13 copies13 the13 file13 using13 the13 ibmcloud13 user13 to13 the13 homeibmcloud13 directory13 on13 the13 virtual13 server13
Install13 and13 access13 the13 virtual13 server13 via13 VNC13 server13 Using13 the13 ssh13 command13 to13 access13 the13 virtual13 server13 is13 not13 the13 most13 stable13 method13 and13 does13 not13 offer13 X-shy‐Windows13 support13 Therefore13 we13 suggest13 that13 you13 use13 a13 VNC13 client13 to13 access13 the13 virtual13 server13
Installing13 a13 VNC13 client13 should13 be13 done13 as13 the13 root13 user13 Use13 the13 yum13 command13 as13 follows13
[ibmcloudelkforcase ~]$ yum groupinstall GNOME Desktop
[ibmcloudelkforcase ~]$ yum install tigervnc-server
These13 commands13 first13 install13 a13 desktop13 style13 OS13 user13 interface13 and13 then13 a13 VNC13 client13 to13 remotely13 access13 the13 desktop13 style13 OS13
Then13 as13 either13 the13 root13 user13 or13 another13 user13 (example13 below13 uses13 elk)13 run13 the13 following13 command13 to13 create13 the13 VNC13 client13 connecon13
[elkelkforcase ~]$ vncserver
New elkforcase2 (elk) desktop is elkforcase2
Starting applications specified in homeelkvncxstartup
Log file is homeelkvncelkforcase2log
[elkelkforcase ~]$
Install13 a13 VNC13 client13 on13 the13 client13 machine13 used13 to13 access13 the13 Internet13 and13 (in13 the13 example13 above)13 use13 the13 command13 ltpublicIPadddressgt213 and13 the13 password13 for13 ELK13 to13 access13 the13 virtual13 server13
Using13 a13 VNC13 client13 means13 that13 you13 can13 leave13 long13 duraon13 installaons13 running13 without13 fear13 of13 the13 terminal13 losing13 connecon13 and13 the13 installaons13 failing13
Install13 the13 ELK13 stack13 Download13 the13 source13 soaware13 code13 for13 the13 ELK13 stack13 from13 the13 ELK13 website13 13
At13 the13 me13 of13 wring13 the13 downloaded13 soaware13 files13 were13
Page of 13 71 Page of 13 71
13
Transfer13 the13 files13 to13 the13 virtual13 server13 (using13 the13 scp13 method13 detailed13 above)13
Install13 Elas0csearch13 As13 a13 root13 user13 create13 a13 non-shy‐root13 user13 with13 the13 adduser13 command13
Create13 a13 directory13 to13 install13 Elascsearch13 into13 and13 untar13 the13 contents13 of13 the13 Elascsearch13 install13 file13 into13 that13 directory13
[elkelkforcase]$ cd home
[elkelkforcase]$ mkdir elasticsearch
[elkelkforcase]$ cp homeibmcloudelasticsearch-233targz homeelasticsearch
[elkelkforcase]$ cd elasticsearch
[elkelkforcase]$ gunzip elasticsearch-233targz
[elkelkforcase]$ untar elasticsearch-233tar
Elascsearch13 is13 now13 installed13
Start13 Elas0csearch13 To13 start13 Elascsearch13 go13 to13 the13 bin13 directory13 as13 the13 non-shy‐root13 user13 when13 you13 installed13 it13 and13 run13 the13 elascsearch13 executable13
[elkelkforcase]$ cd homeelasticsearchelasticsearch-233bin
[elkelkforcase]$ elasticsearch amp
[2] 22211
[elkelkforcase]$ [2016-07-05 133857730][INFO ][node] [Arsenal] version[233] pid[22211] build[218bdf12016-05-17T154004Z]
[2016-07-05 133857740][INFO ][node] [Arsenal] initializing
[2016-07-05 133858984][INFO ][plugins] [Arsenal] modules [reindex lang-expression lang-groovy] plugins [] sites []
[2016-07-05 133859146][INFO ][env] [Arsenal] using [1] data paths mounts [[ (rootfs)]] net usable_space [155gb] net total_space [199gb] spins [unknown] types [rootfs]
[2016-07-05 133859146][INFO ][env] [Arsenal] heap size [10156mb] compressed ordinary object pointers [true]
Page of 14 71 Page of 14 71
[2016-07-05 133859147][WARN ][env] [Arsenal] max file descriptors [4096] for elasticsearch process likely too low consider increasing to at least [65536]
[2016-07-05 133903526][INFO ][node] [Arsenal] initialized
[2016-07-05 133903526][INFO ][node] [Arsenal] starting
To13 test13 that13 Elascsearch13 is13 running13 correctly13 open13 the13 browser13 in13 the13 VNC13 client13 and13 run13 localhost920013
13
Install13 Kibana13 ELK13 recommends13 installing13 and13 running13 Kibana13 as13 root13 13
1 As13 root13 user13 create13 an13 installaon13 directory13 and13 untar13 the13 installaon13 file13
[rootelkforcase]$ cd home
[rootelkforcase]$ mkdir kibana
[rootelkforcase]$ cp homeibmcloudkibana-451-linux-x64targz homekibana
[rootelkforcase]$ cd kibana
[rootelkforcase]$ gunzip kibana-451-linux-x64targz
[rootelkforcase]$ untar kibana-451-linux-x64tar
2 Modify13 the13 manifestyml13 configuraon13 file13 Go13 to13 the13 kibana-shy‐451-shy‐linux-shy‐x6413 directory13
[rootelkforcase]$ cd homekibanakibana-451-linux-x64
3 Edit13 the13 manifestyml13 file13 so13 the13 contents13 are13
applicaons13
-shy‐ name13 kibana-shy‐in-shy‐bluemix13
memory13 128M13
host13 kibana-shy‐in-shy‐bluemix13
buildpack13 hcpsgithubcomcloudfoundry-shy‐communitynginx-shy‐buildpackgit13
Page of 15 71 Page of 15 71
4 Edit13 the13 configkibanayml13 file13 so13 the13 contents13 are13
Kibana is served by a back end server This controls which port to use
port 5601
The host to bind the server to
host 0000
If you are running kibana behind a proxy and want to mount it at a path
specify that path here The basePath cant end in a slash
serverbasePath
The maximum payload size in bytes on incoming server requests
servermaxPayloadBytes 1048576
The Elasticsearch instance to use for all your queries
elasticsearchurl httplocalhost9200
preserve_elasticsearch_host true will send the hostname specified in `elasticsearch` If you set it to false
then the host you use to connect to this Kibana instance will be sent
elasticsearchpreserveHost true
Kibana uses an index in Elasticsearch to store saved searches visualizations
and dashboards It will create a new index if it doesnt already exist
kibanaindex kibana
The default application to load
kibanadefaultAppId discover
If your Elasticsearch is protected with basic auth these are the user credentials
used by the Kibana server to perform maintenance on the kibana_index at startup Your
Kibana users will still need to authenticate with Elasticsearch (which is proxied through
the Kibana server)
Page of 16 71 Page of 16 71
elasticsearchusername user
elasticsearchpassword pass
SSL for outgoing requests from the Kibana Server to the browser (PEM formatted)
serversslcert pathtoyourservercrt
serversslkey pathtoyourserverkey
Optional setting to validate that your Elasticsearch backend uses the same key files (PEM formatted)
elasticsearchsslcert pathtoyourclientcrt
elasticsearchsslkey pathtoyourclientkey
If you need to provide a CA certificate for your Elasticsearch instance put
the path of the pem file here
elasticsearchsslca pathtoyourCApem
Set to false to have a complete disregard for the validity of the SSL
certificate
elasticsearchsslverify true
Time in milliseconds to wait for elasticsearch to respond to pings defaults to
request_timeout setting
elasticsearchpingTimeout 1500
Time in milliseconds to wait for responses from the back end or elasticsearch
This must be gt 0
elasticsearchrequestTimeout 30000
Time in milliseconds for Elasticsearch to wait for responses from shards
Set to 0 to disable
elasticsearchshardTimeout 0
Time in milliseconds to wait for Elasticsearch at Kibana startup before retrying
elasticsearchstartupTimeout 5000
elasticsearchstartupTimeout 30000
Page of 17 71 Page of 17 71
Set the path to where you would like the process id file to be created
pidfile varrunkibanapid
If you would like to send the log output to a file you can set the path below
loggingdest stdout
Set this to true to suppress all logging output
loggingsilent false
Set this to true to suppress all logging output except for error messages
loggingquiet false
Set this to true to log all events including system usage information and all requests
loggingverbose false
Kibana13 is13 now13 installed13
Start13 Kibana13 To13 start13 Kibana13 go13 to13 the13 bin13 directory13 and13 run13 the13 kibana13 executable13
[rootelkforcase]$
[rootelkforcase]$ cd homekibanakibana-451-linux-x64bin
[rootelkforcase]$ kibana amp
[1] 22283
[rootelkforcase] log [095900542] [info][status][pluginkibana] Status changed from uninitialized to green - Ready
log [095900600] [info][status][pluginelasticsearch] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [095900689] [info][status][pluginkbn_vislib_vis_types] Status changed from uninitialized to green - Ready
log [095900717] [info][status][pluginmarkdown_vis] Status changed from uninitialized to green - Ready
log [095900721] [info][status][pluginmetric_vis] Status changed from uninitialized to green - Ready
log [095900814] [info][status][pluginspyModes] Status changed from uninitialized to green - Ready
log [095900934] [info][status][pluginstatusPage] Status changed from uninitialized to green - Ready
Page of 18 71 Page of 18 71
log [095900941] [info][status][plugintable_vis] Status changed from uninitialized to green - Ready
log [095901067] [info][listening] Server running at http00005601
log [095906083] [info][status][pluginelasticsearch] Status changed from yellow to yellow - No existing Kibana index found
log [095908942] [info][status][pluginelasticsearch] Status changed from yellow to green - Kibana index ready
Kibana13 is13 now13 started13
Broadcast13 Bluemix13 app13 log13 records13 Bluemix13 apps13 (which13 are13 shown13 in13 the13 US13 South13 Bluemix13 region13 of13 the13 System13 Architecture13 diagram)13 will13 produce13 logs13 in13 a13 syslog13 rfc542413 format13 using13 a13 Bluemix13 tool13 called13 Loggregator13
For13 Logstash13 to13 receive13 these13 logs13 each13 Bluemix13 app13 needs13 to13 broadcast13 these13 logs13 To13 do13 that13 run13 the13 following13 commands13 on13 each13 Bluemix13 app13 using13 the13 public13 IP13 address13 of13 the13 ELK13 virtual13 server13 the13 ELK13 tag13 to13 trace13 the13 process13 and13 insert13 the13 name13 of13 each13 app13 in13 turn13
cf cups elk -l syslogltpublicIPaddress5000 cf bind-service ltmyappgt elk
cf restage ltmyappgt
Install13 Logstash13 To13 install13 Logstash13 as13 root13 user13 create13 a13 directory13 for13 the13 installaon13 and13 untar13 the13 installaon13 file13 there13
13 13 [rootelkforcase]$ cd home
[rootelkforcase]$ mkdir logstash
[rootelkforcase]$ cp homeibmcloudlogstash-233targz homelogstash
[rootelkforcase]$ cd logstash
[rootelkforcase]$ gunzip logstash-233targz
[rootelkforcase]$ untar logstash-233tar
Write13 Logstash13 configura0on13 Logstash13 requires13 a13 configuraon13 file13 to13 be13 wricen13 to13 receive13 transform13 and13 output13 the13 log13 records13 to13 Elascsearch13 This13 secon13 explains13 the13 basic13 requirements13 of13 processing13 Bluemix13 app13 log13 records13 into13 Elascsearch13 (more13 complex13 configuraons13 are13 detailed13 later13 in13 this13 document13 for13 the13 integraons13 with13 third-shy‐party13 tools)13
Create13 a13 conf13 file13 as13 root13 user13
[rootelkforcase]$ cd homelogstashlogstash-233
[rootelkforcase]$ mkdir config
Page of 19 71 Page of 19 71
[rootelkforcase]$ cd config
[rootelkforcase]$ touch loggregatorconf
Edit13 the13 loggregator13 file13 with13 the13 following13 contents13
13 Input13 secon13 defines13 which13 log13 records13 to13 receive13
13 port13 and13 type13 reflect13 method13 selected13 for13 Cloud13 Foundry13 to13 broadcast13 the13 records13
input
tcp
port =gt 5000
type =gt syslog
13 Filter13 secon13 ndash13 transforms13 the13 data13
filter
Add a field to identify the records as Cloud Foundry based
mutate
add_field =gt format =gt cf
13 Output13 secon13 ndash13 sends13 the13 data13 to13 its13 desnaon(s)13
output
13 Use13 the13 elascsearch13 plugin13 to13 send13 the13 data13 to13 the13 elascsearch13 instance13
13 elasticsearch
hosts =gt localhost
stdout codec =gt rubydebug
13 End13 of13 Config13
Save13 the13 file13 Verify13 the13 format13 of13 the13 file13 is13 acceptable13 by13 running13 the13 following13 commands13
[rootelkforcase]$ cd homelogstashlogstash-233
[rootelkforcase]$ binlogstash agent --configtest -f configloggregatorconf
Page of 20 71 Page of 20 71
Configuraon13 is13 OK13
Start13 Logstash13 Aaer13 you13 have13 wricen13 and13 verified13 the13 Logstash13 configuraon13 you13 can13 start13 Logstash13 to13 process13 records13 from13 the13 Bluemix13 apps13 (via13 Cloud13 Foundry)13 Run13 the13 following13 as13 root13 user13
[rootelkforcase]$ cd homelogstashlogstash-233
[rootelkforcase]$ binlogstash agent -f configloggregatorconf
Logstash13 is13 now13 started13
Monitor13 Logstash13 processing13 Once13 Logstash13 is13 running13 and13 log13 records13 are13 being13 produced13 by13 the13 Bluemix13 apps13 you13 can13 monitor13 progress13 in13 the13 same13 window13 that13 was13 used13 to13 start13 Logstash13 Output13 is13 similar13 to13 the13 following13
13
The13 log13 records13 being13 broadcast13 from13 the13 Bluemix13 apps13 are13 now13 sent13 to13 Elascsearch13 from13 the13 Logstash13 output13 configuraon13 Records13 are13 sent13 in13 real13 me13 If13 Logstash13 is13 not13 running13 while13 log13 records13 are13 being13 produced13 then13 these13 records13 will13 not13 be13 processed13 by13 Logstash13 and13 will13 not13 be13 stored13 in13 Elascsearch13
Page of 21 71 Page of 21 71
Use13 ELK13 The13 log13 records13 from13 the13 Bluemix13 apps13 are13 now13 being13 stored13 and13 indexed13 by13 the13 Elascsearch13 component13 This13 data13 can13 be13 accessed13 via13 the13 Kibana13 applicaon13
Kibana13 Discover13 page13 Using13 a13 web13 browser13 either13 directly13 from13 a13 client13 or13 using13 the13 VNC13 client13 access13 Kibana13 using13 the13 publicIPaddress13 of13 the13 ELK13 virtual13 server13
hcpltpublicIPaddressgt560113
No13 userpassword13 credenals13 are13 required13
Kibana13 defaults13 to13 a13 search13 of13 all13 records13 ()13 in13 the13 last13 1513 minutes13 and13 returns13 the13 results13
13
More13 complex13 searches13 are13 possible13 by13 changing13 the13 me13 filter13 and13 supplying13 search13 queries13
Change13 the13 0me13 filter13 To13 change13 the13 me13 ranges13 click13 the13 clock13 symbol13 in13 the13 top-shy‐right13 of13 the13 Kibana13 Discover13 page13
13
You13 will13 see13 the13 following13 opons13
13
Page of 22 71 Page of 22 71
A13 variety13 of13 quick13 me13 ranges13 can13 be13 selected13 or13 you13 can13 switch13 to13 relave13 or13 absolute13 me13 ranges13
Quick13 me13 ranges13 allow13 switching13 between13 last13 X13 minutes13 to13 last13 Y13 hours13 or13 Z13 days13 etc13
Relave13 me13 ranges13 allow13 more13 specific13 versions13 of13 the13 Quick13 opons13 from13 specific13 minuteshoursdays13 to13 now13
13
Absolute13 me13 ranges13 specify13 the13 start13 and13 end13 mes13 for13 the13 searches13
13
Custom13 search13 queries13 You13 can13 use13 the13 text13 window13 of13 the13 Discover13 page13 to13 search13 for13 matches13 within13 the13 log13 records13 You13 can13 use13 Apache13 Lucene13 or13 Elascsearch13 query13 DSL13 syntax13
13
Page of 23 71 Page of 23 71
Saved13 searches13 You13 can13 save13 any13 custom13 search13 syntax13 and13 choice13 of13 me13 range13 for13 future13 ease13 of13 use13 Any13 relave13 or13 absolute13 me13 frame13 is13 saved13 as13 part13 of13 the13 search13 13
To13 save13 a13 search13
1 Set13 up13 and13 run13 the13 required13 syntax13 and13 me13 frame13
13
2 Click13 the13 Save13 Search13 icon13
13
3 Type13 in13 a13 name13 for13 the13 saved13 search13
Page of 24 71 Page of 24 71
13
7 To13 delete13 or13 update13 saved13 searches13 click13 13 manage13 saved13 searches13
13
Use13 the13 edit13 icons13 to13 make13 any13 changes13
Page of 26 71 Page of 26 71
13
Build13 simple13 Kibana13 histograms13 Kibana13 allows13 you13 to13 build13 charts13 and13 dashboards13 based13 on13 the13 search13 capability13 offered13 by13 the13 Discover13 page13 Saved13 searches13 can13 be13 used13 as13 the13 basis13 for13 charts13 and13 dashboards13 built13 using13 those13 charts13 To13 build13 a13 simple13 histogram13
1 Select13 the13 Visualize13 page13
13
2 Select13 Ver0cal13 Bar13 Chart13
13
3 To13 use13 a13 saved13 search13 click13 From13 a13 saved13 search13
Page of 27 71 Page of 27 71
13
4 Select13 a13 previously13 saved13 search13
13
5 Select13 to13 edit13 the13 x-shy‐axis
6 Edit13 the13 fields13 with13 informaon13 about13 the13 aggregaon13 field13 interval13 and13 custom13 label13
Page of 28 71 Page of 28 71
13
Page of 29 71 Page of 29 71
7 Click13 the13 green13 arrow13 The13 histogram13 is13 created13 and13 run13 based13 on13 the13 saved13 search
13
8 To13 save13 the13 histogram13 click13 the13 Save13 Visualiza0on13 icon13 and13 add13 a13 tle13
13
13
9 Click13 Save13
10 To13 view13 the13 visualisaon13 click13 the13 Dashboard13 page13
Page of 30 71 Page of 30 71
13
11 Click13 the13 Add13 Visualiza0on13 icon13
13
12 Select13 the13 visiualizaon13 that13 you13 just13 created13
13
Page of 31 71 Page of 31 71
13
13 To13 save13 the13 dashboard13 click13 the13 Save13 Dashboard13 icon13
Page of 32 71 Page of 32 71
13
14 Add13 a13 tle13 and13 select13 whether13 to13 store13 the13 selected13 me13 frame13 with13 the13 dashboard13
13
15 Click13 Save13
16 You13 can13 edit13 the13 shape13 and13 number13 of13 charts13 and13 save13 the13 dashboard13
17 To13 re-shy‐run13 the13 dashboard13 select13 the13 Dashboard13 page13 and13 click13 the13 Load13 Saved13 Dashboard13 icon13
Page of 33 71 Page of 33 71
13
18 Select13 the13 saved13 dashboard13 from13 the13 list13
13
19 The13 dashboard13 runs13
13
Page of 34 71 Page of 34 71
Page of 35 71 Page of 35 71
Aler0ng13 with13 ELK13 ELK13 examines13 the13 log13 records13 in13 process13 to13 send13 alerts13 to13 third-shy‐party13 alerng13 event13 management13 and13 social13 media13 applicaons13 As13 shown13 previously13 in13 the13 ELK13 architecture13 the13 ELK13 strategy13 for13 alerng13 is13 to13 send13 alerts13 from13 the13 Logstash13 component13 directly13 Therefore13 log13 records13 and13 alerts13 are13 sent13 to13 both13 Elascsearch13 and13 third-shy‐party13 tools13 at13 the13 same13 instant13 using13 the13 output13 secon13 of13 the13 Logstash13 configuraon13 Any13 alerts13 are13 generated13 during13 the13 filter13 secon13 processing13 of13 the13 Logstash13 configuraon13
Create13 alerts13 in13 Logstash13 The13 Logstash13 filter13 secon13 is13 used13 to13 trigger13 alerts13 based13 on13 the13 log13 records13 being13 processed13 For13 example13 log13 records13 can13 be13 checked13 for13 errors13 warnings13 crashed13 or13 stopped13 messages13 and13 generate13 an13 alert13
Here13 is13 an13 example13 Logstash13 filter13 code13
13 if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
The13 specific13 tags13 are13 set13 against13 the13 log13 records13 to13 trigger13 the13 alert13 later13 in13 the13 configuraon13 in13 the13 output13 secon13
Control13 the13 number13 of13 generated13 alerts13 using13 the13 Logstash13 throcle13 plugin13 so13 that13 either13 a13 lot13 of13 instances13 in13 a13 fixed13 me13 period13 are13 required13 before13 an13 alert13 is13 sent13 or13 a13 specific13 number13 of13 instances13 triggers13 the13 alert13
Page of 36 71 Page of 36 71
Edit13 the13 plugin13 parameters13 to13 meet13 system13 requirements13
13 13 13 13 13 if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 10
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle13
after_count =gt 20
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 30
period =gt 1200
key =gt servicename_alarm_warning
Page of 37 71 Page of 37 71
add_tag =gt servicename_throttled
13 13 13 13 13 13 13 13 13
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 40
period =gt 1500
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
Enrich13 alerts13 with13 message13 detail13 Some13 of13 the13 third-shy‐party13 integraons13 benefit13 from13 having13 components13 of13 the13 raw13 log13 record13 sent13 in13 addion13 to13 the13 raw13 record13 or13 fields13 in13 the13 record13 edited13 transformed13 or13 translated13
Grok13 Logstash13 allows13 the13 log13 record13 to13 be13 broken13 up13 into13 logical13 secons13 and13 each13 secon13 stored13 in13 a13 new13 parameter13 using13 the13 grok13 parameter13 in13 the13 filter13 secon13 Here13 is13 an13 example13 grok13
13 grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
To13 build13 the13 grok13 statement13 Logstash13 offers13 an13 online13 tool13 at13 hcpsgrokdebugherokuappcom13
Translate13 If13 specific13 fields13 in13 the13 log13 record13 are13 coded13 (for13 example13 the13 Bluemix13 app13 name13 is13 oaen13 not13 sent13 in13 the13 syslog13 messages13 instead13 the13 GUID13 is13 sent13 which13 is13 not13 useful)13 then13 the13 filter13 secon13 translate13 plugin13 can13 be13 used13 to13 add13 new13 fields13 by13 holding13 the13 decoded13 versions13 of13 those13 values13
13 translate
field =gt data1
destination =gt data1
override =gt true
Page of 38 71 Page of 38 71
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
dc50f6e4-c052-436a-bf27-95ba392c4228 monverifyservicemanagement
0c677e82-5bfc-4e38-9155-e416249e3ee6 ms-catalog-postpyretic-pomiculture
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
However13 the13 translate13 plugin13 is13 not13 included13 with13 the13 Logstash13 installaon13 In13 the13 next13 secon13 I13 will13 13 describe13 how13 to13 add13 new13 plugins13 to13 the13 Logstash13 instance13
Add13 custom13 plugins13 to13 Logstash13 As13 an13 open13 source13 tool13 Logstash13 encourages13 users13 to13 develop13 content13 for13 all13 users13 to13 share13 Some13 of13 the13 plugins13 that13 were13 created13 by13 our13 community13 include13 a13 translaon13 plugin13 an13 output13 Slack13 plugin13 and13 an13 output13 syslog13 plugin13 Letrsquos13 look13 at13 how13 to13 add13 these13 plugins13 to13 your13 environment13
Filter13 translate13 plugin13 To13 add13 the13 translate13 plugin13
1 Download13 the13 plugin13
2 Copy13 the13 plugin13 file13 logstash-shy‐filter-shy‐translate-shy‐214gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-filter-translate 214
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Output13 Slack13 plugin13 To13 add13 the13 Slack13 plugin13
Page of 39 71 Page of 39 71
1 Download13 the13 plugin13 13
2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐slack-shy‐014gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-output-slack 014
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Output13 syslog13 plugin13 To13 add13 the13 syslog13 plugin13
1 Download13 the13 plugin13 13
2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐syslog-shy‐214gem13 to13 the13 ELK13 Virtual13 Server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-output-syslog 214
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Integrate13 Logstash13 with13 Slack13 To13 send13 generated13 alerts13 to13 Slack13 complete13 these13 steps13
1 Create13 a13 Slack13 integraon13 13
2 Use13 the13 Slack13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 Slack13 integraon13
113 Login13 to13 Slack13 and13 go13 to13 the13 Apps13 page13 13
Page of 40 71 Page of 40 71
13
313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13
13
A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13
13
Page of 42 71 Page of 42 71
The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13
13
The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13
13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
slack
url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5
channel =gt cloud-operations-mon
username =gt nick_cawood
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13
Page of 43 71 Page of 43 71
Page of 44 71 Page of 44 71
In13 Slack13 the13 alerts13 look13 like13 this13
13
Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13
1 Create13 a13 PagerDuty13 integraon13 13
2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13
1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13
13
2 Click13 Add13 New13 Service13
13
313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13
Page of 45 71 Page of 45 71
13
413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13
513 Click13 Add13 Service13
13
The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13
Page of 46 71 Page of 46 71
13
The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt be7eb1ea9c224018923e818d3e891d0b
incident_key =gt logstashhosttype
Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13
Page of 47 71 Page of 47 71
13
Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13
1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13
2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13
3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13
4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13
13 13 http
url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo
http_method =gt post
format =gt json
For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13
13 13 syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
Page of 48 71 Page of 48 71
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13
13
The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13
if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2
This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13
Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13
Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13
Logstash13 severity Shown13 in13 Omnibus13 as
alert Indeterminate
crical Warning
debug 7
emergency Clear
error Minor
informaonal 6
noce Crical
warning Major
Page of 49 71 Page of 49 71
13
Page of 50 71 Page of 50 71
Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13
input
tcp
port =gt 5000
type =gt syslog
filter
mutate
add_field =gt format =gt cf
grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
translate
field =gt data1
destination =gt data1
override =gt true
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
=== Check Alarm Criteria ===
Standard Key Words
if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
=== Alarm Exclusions ===
if [message] =~ (i)sometexthere
mutate remove_tag =gt servicename_alarm
Throttle alarms so we do not hit PD rate limits
Page of 52 71 Page of 52 71
if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 5
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle
after_count =gt 5
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 5
period =gt 1200
key =gt servicename_alarm_warning
add_tag =gt servicename_throttled
Page of 53 71 Page of 53 71
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
output
elasticsearch
hosts =gt localhost
stdout codec =gt rubydebug
if servicename_alarm_err in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
Page of 54 71 Page of 54 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 55 71 Page of 55 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
Page of 56 71 Page of 56 71
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert err message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_error in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
Page of 57 71 Page of 57 71
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 58 71 Page of 58 71
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 59 71 Page of 59 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert error message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
Page of 60 71 Page of 60 71
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
Page of 61 71 Page of 61 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 62 71 Page of 62 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt x
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert fatal message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
Page of 63 71 Page of 63 71
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
Page of 64 71 Page of 64 71
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 65 71 Page of 65 71
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt heavy_exclamation_mark
Page of 66 71 Page of 66 71
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert WARNING message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 67 71 Page of 67 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
Page of 68 71 Page of 68 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 69 71 Page of 69 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
Page of 70 71 Page of 70 71
Page of 71 71 Page of 71 71
13
7 Click13 OK13 Then13 click13 CREATE13
The13 virtual13 server13 is13 provisioned13 and13 added13 to13 the13 Bluemix13 dashboard13 under13 VIRTUAL13 SERVERS13
Page of 11 71 Page of 11 71
13
Set13 up13 the13 virtual13 server13
Log13 on13 to13 the13 virtual13 server13 Now13 you13 can13 log13 on13 to13 the13 virtual13 server13 and13 add13 new13 users13 and13 programs13 to13 make13 it13 easier13 to13 access13 the13 virtual13 server13
To13 access13 the13 server13 go13 to13 the13 UNIX13 or13 Linux13 window13 that13 you13 used13 to13 create13 the13 SSH13 key13 In13 the13 directory13 where13 the13 ltuniquekeynamegt13 files13 exist13 run13 the13 following13 command13 referencing13 the13 SSH13 keyname13 the13 public13 IP13 address13 Bluemix13 allocated13 for13 the13 virtual13 server13 and13 the13 password13 for13 the13 SSH13 key13
[userserver ~]$ ssh -i ltuniquekeynamegt ibmcloudltpublicIPaddressgt
Enter13 the13 passphrase13 for13 the13 key13 ltuniquekeynamegt13
Last login Tue Jul 5 125546 2016 from 108-210-96-231lightspeedrlghncsbcglobalnet
[ibmcloudelkforcase ~]$
ibmcloud13 is13 the13 default13 user13 for13 the13 Bluemix13 virtual13 servers13 and13 allows13 SSH13 to13 access13 the13 server13
Acces13 the13 root13 user13 To13 access13 the13 root13 user13 log13 on13 as13 ibmcloud13 user13 and13 run13 the13 following13 command13
[ibmcloudelkforcase ~]$ sudo bash
[rootelkforcase ibmcloud]
The13 root13 user13 can13 be13 used13 to13 add13 programs13 to13 the13 server13 and13 create13 and13 edit13 users13 as13 required13 to13 access13 applicaons13 installed13 on13 the13 virtual13 server13
Page of 12 71 Page of 12 71
Copy13 files13 to13 the13 virtual13 server13 To13 move13 installaon13 files13 to13 the13 server13 go13 to13 the13 UNIX13 or13 Linux13 window13 you13 used13 to13 create13 the13 SSH13 key13 and13 the13 directory13 where13 the13 ltuniquekeynamegt13 files13 exist13 Move13 the13 installaon13 files13 to13 the13 same13 locaon13
Run13 the13 following13 command13 referencing13 the13 SSH13 keyname13 the13 public13 IP13 address13 Bluemix13 allocated13 for13 the13 virtual13 server13 and13 the13 password13 for13 the13 SSH13 key13
[userserver ~]$ scp ndashi ltuniquekeynamegt ltINSTALLATIONFILENAMEgt ibmcloudltpublicIPaddressgthomeibmcloud
Enter13 the13 passphrase13 for13 key13 cloudkey313
This13 copies13 the13 file13 using13 the13 ibmcloud13 user13 to13 the13 homeibmcloud13 directory13 on13 the13 virtual13 server13
Install13 and13 access13 the13 virtual13 server13 via13 VNC13 server13 Using13 the13 ssh13 command13 to13 access13 the13 virtual13 server13 is13 not13 the13 most13 stable13 method13 and13 does13 not13 offer13 X-shy‐Windows13 support13 Therefore13 we13 suggest13 that13 you13 use13 a13 VNC13 client13 to13 access13 the13 virtual13 server13
Installing13 a13 VNC13 client13 should13 be13 done13 as13 the13 root13 user13 Use13 the13 yum13 command13 as13 follows13
[ibmcloudelkforcase ~]$ yum groupinstall GNOME Desktop
[ibmcloudelkforcase ~]$ yum install tigervnc-server
These13 commands13 first13 install13 a13 desktop13 style13 OS13 user13 interface13 and13 then13 a13 VNC13 client13 to13 remotely13 access13 the13 desktop13 style13 OS13
Then13 as13 either13 the13 root13 user13 or13 another13 user13 (example13 below13 uses13 elk)13 run13 the13 following13 command13 to13 create13 the13 VNC13 client13 connecon13
[elkelkforcase ~]$ vncserver
New elkforcase2 (elk) desktop is elkforcase2
Starting applications specified in homeelkvncxstartup
Log file is homeelkvncelkforcase2log
[elkelkforcase ~]$
Install13 a13 VNC13 client13 on13 the13 client13 machine13 used13 to13 access13 the13 Internet13 and13 (in13 the13 example13 above)13 use13 the13 command13 ltpublicIPadddressgt213 and13 the13 password13 for13 ELK13 to13 access13 the13 virtual13 server13
Using13 a13 VNC13 client13 means13 that13 you13 can13 leave13 long13 duraon13 installaons13 running13 without13 fear13 of13 the13 terminal13 losing13 connecon13 and13 the13 installaons13 failing13
Install13 the13 ELK13 stack13 Download13 the13 source13 soaware13 code13 for13 the13 ELK13 stack13 from13 the13 ELK13 website13 13
At13 the13 me13 of13 wring13 the13 downloaded13 soaware13 files13 were13
Page of 13 71 Page of 13 71
13
Transfer13 the13 files13 to13 the13 virtual13 server13 (using13 the13 scp13 method13 detailed13 above)13
Install13 Elas0csearch13 As13 a13 root13 user13 create13 a13 non-shy‐root13 user13 with13 the13 adduser13 command13
Create13 a13 directory13 to13 install13 Elascsearch13 into13 and13 untar13 the13 contents13 of13 the13 Elascsearch13 install13 file13 into13 that13 directory13
[elkelkforcase]$ cd home
[elkelkforcase]$ mkdir elasticsearch
[elkelkforcase]$ cp homeibmcloudelasticsearch-233targz homeelasticsearch
[elkelkforcase]$ cd elasticsearch
[elkelkforcase]$ gunzip elasticsearch-233targz
[elkelkforcase]$ untar elasticsearch-233tar
Elascsearch13 is13 now13 installed13
Start13 Elas0csearch13 To13 start13 Elascsearch13 go13 to13 the13 bin13 directory13 as13 the13 non-shy‐root13 user13 when13 you13 installed13 it13 and13 run13 the13 elascsearch13 executable13
[elkelkforcase]$ cd homeelasticsearchelasticsearch-233bin
[elkelkforcase]$ elasticsearch amp
[2] 22211
[elkelkforcase]$ [2016-07-05 133857730][INFO ][node] [Arsenal] version[233] pid[22211] build[218bdf12016-05-17T154004Z]
[2016-07-05 133857740][INFO ][node] [Arsenal] initializing
[2016-07-05 133858984][INFO ][plugins] [Arsenal] modules [reindex lang-expression lang-groovy] plugins [] sites []
[2016-07-05 133859146][INFO ][env] [Arsenal] using [1] data paths mounts [[ (rootfs)]] net usable_space [155gb] net total_space [199gb] spins [unknown] types [rootfs]
[2016-07-05 133859146][INFO ][env] [Arsenal] heap size [10156mb] compressed ordinary object pointers [true]
Page of 14 71 Page of 14 71
[2016-07-05 133859147][WARN ][env] [Arsenal] max file descriptors [4096] for elasticsearch process likely too low consider increasing to at least [65536]
[2016-07-05 133903526][INFO ][node] [Arsenal] initialized
[2016-07-05 133903526][INFO ][node] [Arsenal] starting
To13 test13 that13 Elascsearch13 is13 running13 correctly13 open13 the13 browser13 in13 the13 VNC13 client13 and13 run13 localhost920013
13
Install13 Kibana13 ELK13 recommends13 installing13 and13 running13 Kibana13 as13 root13 13
1 As13 root13 user13 create13 an13 installaon13 directory13 and13 untar13 the13 installaon13 file13
[rootelkforcase]$ cd home
[rootelkforcase]$ mkdir kibana
[rootelkforcase]$ cp homeibmcloudkibana-451-linux-x64targz homekibana
[rootelkforcase]$ cd kibana
[rootelkforcase]$ gunzip kibana-451-linux-x64targz
[rootelkforcase]$ untar kibana-451-linux-x64tar
2 Modify13 the13 manifestyml13 configuraon13 file13 Go13 to13 the13 kibana-shy‐451-shy‐linux-shy‐x6413 directory13
[rootelkforcase]$ cd homekibanakibana-451-linux-x64
3 Edit13 the13 manifestyml13 file13 so13 the13 contents13 are13
applicaons13
-shy‐ name13 kibana-shy‐in-shy‐bluemix13
memory13 128M13
host13 kibana-shy‐in-shy‐bluemix13
buildpack13 hcpsgithubcomcloudfoundry-shy‐communitynginx-shy‐buildpackgit13
Page of 15 71 Page of 15 71
4 Edit13 the13 configkibanayml13 file13 so13 the13 contents13 are13
Kibana is served by a back end server This controls which port to use
port 5601
The host to bind the server to
host 0000
If you are running kibana behind a proxy and want to mount it at a path
specify that path here The basePath cant end in a slash
serverbasePath
The maximum payload size in bytes on incoming server requests
servermaxPayloadBytes 1048576
The Elasticsearch instance to use for all your queries
elasticsearchurl httplocalhost9200
preserve_elasticsearch_host true will send the hostname specified in `elasticsearch` If you set it to false
then the host you use to connect to this Kibana instance will be sent
elasticsearchpreserveHost true
Kibana uses an index in Elasticsearch to store saved searches visualizations
and dashboards It will create a new index if it doesnt already exist
kibanaindex kibana
The default application to load
kibanadefaultAppId discover
If your Elasticsearch is protected with basic auth these are the user credentials
used by the Kibana server to perform maintenance on the kibana_index at startup Your
Kibana users will still need to authenticate with Elasticsearch (which is proxied through
the Kibana server)
Page of 16 71 Page of 16 71
elasticsearchusername user
elasticsearchpassword pass
SSL for outgoing requests from the Kibana Server to the browser (PEM formatted)
serversslcert pathtoyourservercrt
serversslkey pathtoyourserverkey
Optional setting to validate that your Elasticsearch backend uses the same key files (PEM formatted)
elasticsearchsslcert pathtoyourclientcrt
elasticsearchsslkey pathtoyourclientkey
If you need to provide a CA certificate for your Elasticsearch instance put
the path of the pem file here
elasticsearchsslca pathtoyourCApem
Set to false to have a complete disregard for the validity of the SSL
certificate
elasticsearchsslverify true
Time in milliseconds to wait for elasticsearch to respond to pings defaults to
request_timeout setting
elasticsearchpingTimeout 1500
Time in milliseconds to wait for responses from the back end or elasticsearch
This must be gt 0
elasticsearchrequestTimeout 30000
Time in milliseconds for Elasticsearch to wait for responses from shards
Set to 0 to disable
elasticsearchshardTimeout 0
Time in milliseconds to wait for Elasticsearch at Kibana startup before retrying
elasticsearchstartupTimeout 5000
elasticsearchstartupTimeout 30000
Page of 17 71 Page of 17 71
Set the path to where you would like the process id file to be created
pidfile varrunkibanapid
If you would like to send the log output to a file you can set the path below
loggingdest stdout
Set this to true to suppress all logging output
loggingsilent false
Set this to true to suppress all logging output except for error messages
loggingquiet false
Set this to true to log all events including system usage information and all requests
loggingverbose false
Kibana13 is13 now13 installed13
Start13 Kibana13 To13 start13 Kibana13 go13 to13 the13 bin13 directory13 and13 run13 the13 kibana13 executable13
[rootelkforcase]$
[rootelkforcase]$ cd homekibanakibana-451-linux-x64bin
[rootelkforcase]$ kibana amp
[1] 22283
[rootelkforcase] log [095900542] [info][status][pluginkibana] Status changed from uninitialized to green - Ready
log [095900600] [info][status][pluginelasticsearch] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [095900689] [info][status][pluginkbn_vislib_vis_types] Status changed from uninitialized to green - Ready
log [095900717] [info][status][pluginmarkdown_vis] Status changed from uninitialized to green - Ready
log [095900721] [info][status][pluginmetric_vis] Status changed from uninitialized to green - Ready
log [095900814] [info][status][pluginspyModes] Status changed from uninitialized to green - Ready
log [095900934] [info][status][pluginstatusPage] Status changed from uninitialized to green - Ready
Page of 18 71 Page of 18 71
log [095900941] [info][status][plugintable_vis] Status changed from uninitialized to green - Ready
log [095901067] [info][listening] Server running at http00005601
log [095906083] [info][status][pluginelasticsearch] Status changed from yellow to yellow - No existing Kibana index found
log [095908942] [info][status][pluginelasticsearch] Status changed from yellow to green - Kibana index ready
Kibana13 is13 now13 started13
Broadcast13 Bluemix13 app13 log13 records13 Bluemix13 apps13 (which13 are13 shown13 in13 the13 US13 South13 Bluemix13 region13 of13 the13 System13 Architecture13 diagram)13 will13 produce13 logs13 in13 a13 syslog13 rfc542413 format13 using13 a13 Bluemix13 tool13 called13 Loggregator13
For13 Logstash13 to13 receive13 these13 logs13 each13 Bluemix13 app13 needs13 to13 broadcast13 these13 logs13 To13 do13 that13 run13 the13 following13 commands13 on13 each13 Bluemix13 app13 using13 the13 public13 IP13 address13 of13 the13 ELK13 virtual13 server13 the13 ELK13 tag13 to13 trace13 the13 process13 and13 insert13 the13 name13 of13 each13 app13 in13 turn13
cf cups elk -l syslogltpublicIPaddress5000 cf bind-service ltmyappgt elk
cf restage ltmyappgt
Install13 Logstash13 To13 install13 Logstash13 as13 root13 user13 create13 a13 directory13 for13 the13 installaon13 and13 untar13 the13 installaon13 file13 there13
13 13 [rootelkforcase]$ cd home
[rootelkforcase]$ mkdir logstash
[rootelkforcase]$ cp homeibmcloudlogstash-233targz homelogstash
[rootelkforcase]$ cd logstash
[rootelkforcase]$ gunzip logstash-233targz
[rootelkforcase]$ untar logstash-233tar
Write13 Logstash13 configura0on13 Logstash13 requires13 a13 configuraon13 file13 to13 be13 wricen13 to13 receive13 transform13 and13 output13 the13 log13 records13 to13 Elascsearch13 This13 secon13 explains13 the13 basic13 requirements13 of13 processing13 Bluemix13 app13 log13 records13 into13 Elascsearch13 (more13 complex13 configuraons13 are13 detailed13 later13 in13 this13 document13 for13 the13 integraons13 with13 third-shy‐party13 tools)13
Create13 a13 conf13 file13 as13 root13 user13
[rootelkforcase]$ cd homelogstashlogstash-233
[rootelkforcase]$ mkdir config
Page of 19 71 Page of 19 71
[rootelkforcase]$ cd config
[rootelkforcase]$ touch loggregatorconf
Edit13 the13 loggregator13 file13 with13 the13 following13 contents13
13 Input13 secon13 defines13 which13 log13 records13 to13 receive13
13 port13 and13 type13 reflect13 method13 selected13 for13 Cloud13 Foundry13 to13 broadcast13 the13 records13
input
tcp
port =gt 5000
type =gt syslog
13 Filter13 secon13 ndash13 transforms13 the13 data13
filter
Add a field to identify the records as Cloud Foundry based
mutate
add_field =gt format =gt cf
13 Output13 secon13 ndash13 sends13 the13 data13 to13 its13 desnaon(s)13
output
13 Use13 the13 elascsearch13 plugin13 to13 send13 the13 data13 to13 the13 elascsearch13 instance13
13 elasticsearch
hosts =gt localhost
stdout codec =gt rubydebug
13 End13 of13 Config13
Save13 the13 file13 Verify13 the13 format13 of13 the13 file13 is13 acceptable13 by13 running13 the13 following13 commands13
[rootelkforcase]$ cd homelogstashlogstash-233
[rootelkforcase]$ binlogstash agent --configtest -f configloggregatorconf
Page of 20 71 Page of 20 71
Configuraon13 is13 OK13
Start13 Logstash13 Aaer13 you13 have13 wricen13 and13 verified13 the13 Logstash13 configuraon13 you13 can13 start13 Logstash13 to13 process13 records13 from13 the13 Bluemix13 apps13 (via13 Cloud13 Foundry)13 Run13 the13 following13 as13 root13 user13
[rootelkforcase]$ cd homelogstashlogstash-233
[rootelkforcase]$ binlogstash agent -f configloggregatorconf
Logstash13 is13 now13 started13
Monitor13 Logstash13 processing13 Once13 Logstash13 is13 running13 and13 log13 records13 are13 being13 produced13 by13 the13 Bluemix13 apps13 you13 can13 monitor13 progress13 in13 the13 same13 window13 that13 was13 used13 to13 start13 Logstash13 Output13 is13 similar13 to13 the13 following13
13
The13 log13 records13 being13 broadcast13 from13 the13 Bluemix13 apps13 are13 now13 sent13 to13 Elascsearch13 from13 the13 Logstash13 output13 configuraon13 Records13 are13 sent13 in13 real13 me13 If13 Logstash13 is13 not13 running13 while13 log13 records13 are13 being13 produced13 then13 these13 records13 will13 not13 be13 processed13 by13 Logstash13 and13 will13 not13 be13 stored13 in13 Elascsearch13
Page of 21 71 Page of 21 71
Use13 ELK13 The13 log13 records13 from13 the13 Bluemix13 apps13 are13 now13 being13 stored13 and13 indexed13 by13 the13 Elascsearch13 component13 This13 data13 can13 be13 accessed13 via13 the13 Kibana13 applicaon13
Kibana13 Discover13 page13 Using13 a13 web13 browser13 either13 directly13 from13 a13 client13 or13 using13 the13 VNC13 client13 access13 Kibana13 using13 the13 publicIPaddress13 of13 the13 ELK13 virtual13 server13
hcpltpublicIPaddressgt560113
No13 userpassword13 credenals13 are13 required13
Kibana13 defaults13 to13 a13 search13 of13 all13 records13 ()13 in13 the13 last13 1513 minutes13 and13 returns13 the13 results13
13
More13 complex13 searches13 are13 possible13 by13 changing13 the13 me13 filter13 and13 supplying13 search13 queries13
Change13 the13 0me13 filter13 To13 change13 the13 me13 ranges13 click13 the13 clock13 symbol13 in13 the13 top-shy‐right13 of13 the13 Kibana13 Discover13 page13
13
You13 will13 see13 the13 following13 opons13
13
Page of 22 71 Page of 22 71
A13 variety13 of13 quick13 me13 ranges13 can13 be13 selected13 or13 you13 can13 switch13 to13 relave13 or13 absolute13 me13 ranges13
Quick13 me13 ranges13 allow13 switching13 between13 last13 X13 minutes13 to13 last13 Y13 hours13 or13 Z13 days13 etc13
Relave13 me13 ranges13 allow13 more13 specific13 versions13 of13 the13 Quick13 opons13 from13 specific13 minuteshoursdays13 to13 now13
13
Absolute13 me13 ranges13 specify13 the13 start13 and13 end13 mes13 for13 the13 searches13
13
Custom13 search13 queries13 You13 can13 use13 the13 text13 window13 of13 the13 Discover13 page13 to13 search13 for13 matches13 within13 the13 log13 records13 You13 can13 use13 Apache13 Lucene13 or13 Elascsearch13 query13 DSL13 syntax13
13
Page of 23 71 Page of 23 71
Saved13 searches13 You13 can13 save13 any13 custom13 search13 syntax13 and13 choice13 of13 me13 range13 for13 future13 ease13 of13 use13 Any13 relave13 or13 absolute13 me13 frame13 is13 saved13 as13 part13 of13 the13 search13 13
To13 save13 a13 search13
1 Set13 up13 and13 run13 the13 required13 syntax13 and13 me13 frame13
13
2 Click13 the13 Save13 Search13 icon13
13
3 Type13 in13 a13 name13 for13 the13 saved13 search13
Page of 24 71 Page of 24 71
13
7 To13 delete13 or13 update13 saved13 searches13 click13 13 manage13 saved13 searches13
13
Use13 the13 edit13 icons13 to13 make13 any13 changes13
Page of 26 71 Page of 26 71
13
Build13 simple13 Kibana13 histograms13 Kibana13 allows13 you13 to13 build13 charts13 and13 dashboards13 based13 on13 the13 search13 capability13 offered13 by13 the13 Discover13 page13 Saved13 searches13 can13 be13 used13 as13 the13 basis13 for13 charts13 and13 dashboards13 built13 using13 those13 charts13 To13 build13 a13 simple13 histogram13
1 Select13 the13 Visualize13 page13
13
2 Select13 Ver0cal13 Bar13 Chart13
13
3 To13 use13 a13 saved13 search13 click13 From13 a13 saved13 search13
Page of 27 71 Page of 27 71
13
4 Select13 a13 previously13 saved13 search13
13
5 Select13 to13 edit13 the13 x-shy‐axis
6 Edit13 the13 fields13 with13 informaon13 about13 the13 aggregaon13 field13 interval13 and13 custom13 label13
Page of 28 71 Page of 28 71
13
Page of 29 71 Page of 29 71
7 Click13 the13 green13 arrow13 The13 histogram13 is13 created13 and13 run13 based13 on13 the13 saved13 search
13
8 To13 save13 the13 histogram13 click13 the13 Save13 Visualiza0on13 icon13 and13 add13 a13 tle13
13
13
9 Click13 Save13
10 To13 view13 the13 visualisaon13 click13 the13 Dashboard13 page13
Page of 30 71 Page of 30 71
13
11 Click13 the13 Add13 Visualiza0on13 icon13
13
12 Select13 the13 visiualizaon13 that13 you13 just13 created13
13
Page of 31 71 Page of 31 71
13
13 To13 save13 the13 dashboard13 click13 the13 Save13 Dashboard13 icon13
Page of 32 71 Page of 32 71
13
14 Add13 a13 tle13 and13 select13 whether13 to13 store13 the13 selected13 me13 frame13 with13 the13 dashboard13
13
15 Click13 Save13
16 You13 can13 edit13 the13 shape13 and13 number13 of13 charts13 and13 save13 the13 dashboard13
17 To13 re-shy‐run13 the13 dashboard13 select13 the13 Dashboard13 page13 and13 click13 the13 Load13 Saved13 Dashboard13 icon13
Page of 33 71 Page of 33 71
13
18 Select13 the13 saved13 dashboard13 from13 the13 list13
13
19 The13 dashboard13 runs13
13
Page of 34 71 Page of 34 71
Page of 35 71 Page of 35 71
Aler0ng13 with13 ELK13 ELK13 examines13 the13 log13 records13 in13 process13 to13 send13 alerts13 to13 third-shy‐party13 alerng13 event13 management13 and13 social13 media13 applicaons13 As13 shown13 previously13 in13 the13 ELK13 architecture13 the13 ELK13 strategy13 for13 alerng13 is13 to13 send13 alerts13 from13 the13 Logstash13 component13 directly13 Therefore13 log13 records13 and13 alerts13 are13 sent13 to13 both13 Elascsearch13 and13 third-shy‐party13 tools13 at13 the13 same13 instant13 using13 the13 output13 secon13 of13 the13 Logstash13 configuraon13 Any13 alerts13 are13 generated13 during13 the13 filter13 secon13 processing13 of13 the13 Logstash13 configuraon13
Create13 alerts13 in13 Logstash13 The13 Logstash13 filter13 secon13 is13 used13 to13 trigger13 alerts13 based13 on13 the13 log13 records13 being13 processed13 For13 example13 log13 records13 can13 be13 checked13 for13 errors13 warnings13 crashed13 or13 stopped13 messages13 and13 generate13 an13 alert13
Here13 is13 an13 example13 Logstash13 filter13 code13
13 if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
The13 specific13 tags13 are13 set13 against13 the13 log13 records13 to13 trigger13 the13 alert13 later13 in13 the13 configuraon13 in13 the13 output13 secon13
Control13 the13 number13 of13 generated13 alerts13 using13 the13 Logstash13 throcle13 plugin13 so13 that13 either13 a13 lot13 of13 instances13 in13 a13 fixed13 me13 period13 are13 required13 before13 an13 alert13 is13 sent13 or13 a13 specific13 number13 of13 instances13 triggers13 the13 alert13
Page of 36 71 Page of 36 71
Edit13 the13 plugin13 parameters13 to13 meet13 system13 requirements13
13 13 13 13 13 if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 10
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle13
after_count =gt 20
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 30
period =gt 1200
key =gt servicename_alarm_warning
Page of 37 71 Page of 37 71
add_tag =gt servicename_throttled
13 13 13 13 13 13 13 13 13
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 40
period =gt 1500
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
Enrich13 alerts13 with13 message13 detail13 Some13 of13 the13 third-shy‐party13 integraons13 benefit13 from13 having13 components13 of13 the13 raw13 log13 record13 sent13 in13 addion13 to13 the13 raw13 record13 or13 fields13 in13 the13 record13 edited13 transformed13 or13 translated13
Grok13 Logstash13 allows13 the13 log13 record13 to13 be13 broken13 up13 into13 logical13 secons13 and13 each13 secon13 stored13 in13 a13 new13 parameter13 using13 the13 grok13 parameter13 in13 the13 filter13 secon13 Here13 is13 an13 example13 grok13
13 grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
To13 build13 the13 grok13 statement13 Logstash13 offers13 an13 online13 tool13 at13 hcpsgrokdebugherokuappcom13
Translate13 If13 specific13 fields13 in13 the13 log13 record13 are13 coded13 (for13 example13 the13 Bluemix13 app13 name13 is13 oaen13 not13 sent13 in13 the13 syslog13 messages13 instead13 the13 GUID13 is13 sent13 which13 is13 not13 useful)13 then13 the13 filter13 secon13 translate13 plugin13 can13 be13 used13 to13 add13 new13 fields13 by13 holding13 the13 decoded13 versions13 of13 those13 values13
13 translate
field =gt data1
destination =gt data1
override =gt true
Page of 38 71 Page of 38 71
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
dc50f6e4-c052-436a-bf27-95ba392c4228 monverifyservicemanagement
0c677e82-5bfc-4e38-9155-e416249e3ee6 ms-catalog-postpyretic-pomiculture
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
However13 the13 translate13 plugin13 is13 not13 included13 with13 the13 Logstash13 installaon13 In13 the13 next13 secon13 I13 will13 13 describe13 how13 to13 add13 new13 plugins13 to13 the13 Logstash13 instance13
Add13 custom13 plugins13 to13 Logstash13 As13 an13 open13 source13 tool13 Logstash13 encourages13 users13 to13 develop13 content13 for13 all13 users13 to13 share13 Some13 of13 the13 plugins13 that13 were13 created13 by13 our13 community13 include13 a13 translaon13 plugin13 an13 output13 Slack13 plugin13 and13 an13 output13 syslog13 plugin13 Letrsquos13 look13 at13 how13 to13 add13 these13 plugins13 to13 your13 environment13
Filter13 translate13 plugin13 To13 add13 the13 translate13 plugin13
1 Download13 the13 plugin13
2 Copy13 the13 plugin13 file13 logstash-shy‐filter-shy‐translate-shy‐214gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-filter-translate 214
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Output13 Slack13 plugin13 To13 add13 the13 Slack13 plugin13
Page of 39 71 Page of 39 71
1 Download13 the13 plugin13 13
2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐slack-shy‐014gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-output-slack 014
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Output13 syslog13 plugin13 To13 add13 the13 syslog13 plugin13
1 Download13 the13 plugin13 13
2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐syslog-shy‐214gem13 to13 the13 ELK13 Virtual13 Server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-output-syslog 214
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Integrate13 Logstash13 with13 Slack13 To13 send13 generated13 alerts13 to13 Slack13 complete13 these13 steps13
1 Create13 a13 Slack13 integraon13 13
2 Use13 the13 Slack13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 Slack13 integraon13
113 Login13 to13 Slack13 and13 go13 to13 the13 Apps13 page13 13
Page of 40 71 Page of 40 71
13
313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13
13
A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13
13
Page of 42 71 Page of 42 71
The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13
13
The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13
13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
slack
url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5
channel =gt cloud-operations-mon
username =gt nick_cawood
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13
Page of 43 71 Page of 43 71
Page of 44 71 Page of 44 71
In13 Slack13 the13 alerts13 look13 like13 this13
13
Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13
1 Create13 a13 PagerDuty13 integraon13 13
2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13
1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13
13
2 Click13 Add13 New13 Service13
13
313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13
Page of 45 71 Page of 45 71
13
413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13
513 Click13 Add13 Service13
13
The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13
Page of 46 71 Page of 46 71
13
The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt be7eb1ea9c224018923e818d3e891d0b
incident_key =gt logstashhosttype
Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13
Page of 47 71 Page of 47 71
13
Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13
1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13
2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13
3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13
4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13
13 13 http
url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo
http_method =gt post
format =gt json
For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13
13 13 syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
Page of 48 71 Page of 48 71
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13
13
The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13
if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2
This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13
Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13
Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13
Logstash13 severity Shown13 in13 Omnibus13 as
alert Indeterminate
crical Warning
debug 7
emergency Clear
error Minor
informaonal 6
noce Crical
warning Major
Page of 49 71 Page of 49 71
13
Page of 50 71 Page of 50 71
Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13
input
tcp
port =gt 5000
type =gt syslog
filter
mutate
add_field =gt format =gt cf
grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
translate
field =gt data1
destination =gt data1
override =gt true
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
=== Check Alarm Criteria ===
Standard Key Words
if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
=== Alarm Exclusions ===
if [message] =~ (i)sometexthere
mutate remove_tag =gt servicename_alarm
Throttle alarms so we do not hit PD rate limits
Page of 52 71 Page of 52 71
if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 5
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle
after_count =gt 5
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 5
period =gt 1200
key =gt servicename_alarm_warning
add_tag =gt servicename_throttled
Page of 53 71 Page of 53 71
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
output
elasticsearch
hosts =gt localhost
stdout codec =gt rubydebug
if servicename_alarm_err in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
Page of 54 71 Page of 54 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 55 71 Page of 55 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
Page of 56 71 Page of 56 71
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert err message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_error in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
Page of 57 71 Page of 57 71
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 58 71 Page of 58 71
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 59 71 Page of 59 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert error message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
Page of 60 71 Page of 60 71
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
Page of 61 71 Page of 61 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 62 71 Page of 62 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt x
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert fatal message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
Page of 63 71 Page of 63 71
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
Page of 64 71 Page of 64 71
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 65 71 Page of 65 71
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt heavy_exclamation_mark
Page of 66 71 Page of 66 71
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert WARNING message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 67 71 Page of 67 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
Page of 68 71 Page of 68 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 69 71 Page of 69 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
Page of 70 71 Page of 70 71
Page of 71 71 Page of 71 71
13
Set13 up13 the13 virtual13 server13
Log13 on13 to13 the13 virtual13 server13 Now13 you13 can13 log13 on13 to13 the13 virtual13 server13 and13 add13 new13 users13 and13 programs13 to13 make13 it13 easier13 to13 access13 the13 virtual13 server13
To13 access13 the13 server13 go13 to13 the13 UNIX13 or13 Linux13 window13 that13 you13 used13 to13 create13 the13 SSH13 key13 In13 the13 directory13 where13 the13 ltuniquekeynamegt13 files13 exist13 run13 the13 following13 command13 referencing13 the13 SSH13 keyname13 the13 public13 IP13 address13 Bluemix13 allocated13 for13 the13 virtual13 server13 and13 the13 password13 for13 the13 SSH13 key13
[userserver ~]$ ssh -i ltuniquekeynamegt ibmcloudltpublicIPaddressgt
Enter13 the13 passphrase13 for13 the13 key13 ltuniquekeynamegt13
Last login Tue Jul 5 125546 2016 from 108-210-96-231lightspeedrlghncsbcglobalnet
[ibmcloudelkforcase ~]$
ibmcloud13 is13 the13 default13 user13 for13 the13 Bluemix13 virtual13 servers13 and13 allows13 SSH13 to13 access13 the13 server13
Acces13 the13 root13 user13 To13 access13 the13 root13 user13 log13 on13 as13 ibmcloud13 user13 and13 run13 the13 following13 command13
[ibmcloudelkforcase ~]$ sudo bash
[rootelkforcase ibmcloud]
The13 root13 user13 can13 be13 used13 to13 add13 programs13 to13 the13 server13 and13 create13 and13 edit13 users13 as13 required13 to13 access13 applicaons13 installed13 on13 the13 virtual13 server13
Page of 12 71 Page of 12 71
Copy13 files13 to13 the13 virtual13 server13 To13 move13 installaon13 files13 to13 the13 server13 go13 to13 the13 UNIX13 or13 Linux13 window13 you13 used13 to13 create13 the13 SSH13 key13 and13 the13 directory13 where13 the13 ltuniquekeynamegt13 files13 exist13 Move13 the13 installaon13 files13 to13 the13 same13 locaon13
Run13 the13 following13 command13 referencing13 the13 SSH13 keyname13 the13 public13 IP13 address13 Bluemix13 allocated13 for13 the13 virtual13 server13 and13 the13 password13 for13 the13 SSH13 key13
[userserver ~]$ scp ndashi ltuniquekeynamegt ltINSTALLATIONFILENAMEgt ibmcloudltpublicIPaddressgthomeibmcloud
Enter13 the13 passphrase13 for13 key13 cloudkey313
This13 copies13 the13 file13 using13 the13 ibmcloud13 user13 to13 the13 homeibmcloud13 directory13 on13 the13 virtual13 server13
Install13 and13 access13 the13 virtual13 server13 via13 VNC13 server13 Using13 the13 ssh13 command13 to13 access13 the13 virtual13 server13 is13 not13 the13 most13 stable13 method13 and13 does13 not13 offer13 X-shy‐Windows13 support13 Therefore13 we13 suggest13 that13 you13 use13 a13 VNC13 client13 to13 access13 the13 virtual13 server13
Installing13 a13 VNC13 client13 should13 be13 done13 as13 the13 root13 user13 Use13 the13 yum13 command13 as13 follows13
[ibmcloudelkforcase ~]$ yum groupinstall GNOME Desktop
[ibmcloudelkforcase ~]$ yum install tigervnc-server
These13 commands13 first13 install13 a13 desktop13 style13 OS13 user13 interface13 and13 then13 a13 VNC13 client13 to13 remotely13 access13 the13 desktop13 style13 OS13
Then13 as13 either13 the13 root13 user13 or13 another13 user13 (example13 below13 uses13 elk)13 run13 the13 following13 command13 to13 create13 the13 VNC13 client13 connecon13
[elkelkforcase ~]$ vncserver
New elkforcase2 (elk) desktop is elkforcase2
Starting applications specified in homeelkvncxstartup
Log file is homeelkvncelkforcase2log
[elkelkforcase ~]$
Install13 a13 VNC13 client13 on13 the13 client13 machine13 used13 to13 access13 the13 Internet13 and13 (in13 the13 example13 above)13 use13 the13 command13 ltpublicIPadddressgt213 and13 the13 password13 for13 ELK13 to13 access13 the13 virtual13 server13
Using13 a13 VNC13 client13 means13 that13 you13 can13 leave13 long13 duraon13 installaons13 running13 without13 fear13 of13 the13 terminal13 losing13 connecon13 and13 the13 installaons13 failing13
Install13 the13 ELK13 stack13 Download13 the13 source13 soaware13 code13 for13 the13 ELK13 stack13 from13 the13 ELK13 website13 13
At13 the13 me13 of13 wring13 the13 downloaded13 soaware13 files13 were13
Page of 13 71 Page of 13 71
13
Transfer13 the13 files13 to13 the13 virtual13 server13 (using13 the13 scp13 method13 detailed13 above)13
Install13 Elas0csearch13 As13 a13 root13 user13 create13 a13 non-shy‐root13 user13 with13 the13 adduser13 command13
Create13 a13 directory13 to13 install13 Elascsearch13 into13 and13 untar13 the13 contents13 of13 the13 Elascsearch13 install13 file13 into13 that13 directory13
[elkelkforcase]$ cd home
[elkelkforcase]$ mkdir elasticsearch
[elkelkforcase]$ cp homeibmcloudelasticsearch-233targz homeelasticsearch
[elkelkforcase]$ cd elasticsearch
[elkelkforcase]$ gunzip elasticsearch-233targz
[elkelkforcase]$ untar elasticsearch-233tar
Elascsearch13 is13 now13 installed13
Start13 Elas0csearch13 To13 start13 Elascsearch13 go13 to13 the13 bin13 directory13 as13 the13 non-shy‐root13 user13 when13 you13 installed13 it13 and13 run13 the13 elascsearch13 executable13
[elkelkforcase]$ cd homeelasticsearchelasticsearch-233bin
[elkelkforcase]$ elasticsearch amp
[2] 22211
[elkelkforcase]$ [2016-07-05 133857730][INFO ][node] [Arsenal] version[233] pid[22211] build[218bdf12016-05-17T154004Z]
[2016-07-05 133857740][INFO ][node] [Arsenal] initializing
[2016-07-05 133858984][INFO ][plugins] [Arsenal] modules [reindex lang-expression lang-groovy] plugins [] sites []
[2016-07-05 133859146][INFO ][env] [Arsenal] using [1] data paths mounts [[ (rootfs)]] net usable_space [155gb] net total_space [199gb] spins [unknown] types [rootfs]
[2016-07-05 133859146][INFO ][env] [Arsenal] heap size [10156mb] compressed ordinary object pointers [true]
Page of 14 71 Page of 14 71
[2016-07-05 133859147][WARN ][env] [Arsenal] max file descriptors [4096] for elasticsearch process likely too low consider increasing to at least [65536]
[2016-07-05 133903526][INFO ][node] [Arsenal] initialized
[2016-07-05 133903526][INFO ][node] [Arsenal] starting
To13 test13 that13 Elascsearch13 is13 running13 correctly13 open13 the13 browser13 in13 the13 VNC13 client13 and13 run13 localhost920013
13
Install13 Kibana13 ELK13 recommends13 installing13 and13 running13 Kibana13 as13 root13 13
1 As13 root13 user13 create13 an13 installaon13 directory13 and13 untar13 the13 installaon13 file13
[rootelkforcase]$ cd home
[rootelkforcase]$ mkdir kibana
[rootelkforcase]$ cp homeibmcloudkibana-451-linux-x64targz homekibana
[rootelkforcase]$ cd kibana
[rootelkforcase]$ gunzip kibana-451-linux-x64targz
[rootelkforcase]$ untar kibana-451-linux-x64tar
2 Modify13 the13 manifestyml13 configuraon13 file13 Go13 to13 the13 kibana-shy‐451-shy‐linux-shy‐x6413 directory13
[rootelkforcase]$ cd homekibanakibana-451-linux-x64
3 Edit13 the13 manifestyml13 file13 so13 the13 contents13 are13
applicaons13
-shy‐ name13 kibana-shy‐in-shy‐bluemix13
memory13 128M13
host13 kibana-shy‐in-shy‐bluemix13
buildpack13 hcpsgithubcomcloudfoundry-shy‐communitynginx-shy‐buildpackgit13
Page of 15 71 Page of 15 71
4 Edit13 the13 configkibanayml13 file13 so13 the13 contents13 are13
Kibana is served by a back end server This controls which port to use
port 5601
The host to bind the server to
host 0000
If you are running kibana behind a proxy and want to mount it at a path
specify that path here The basePath cant end in a slash
serverbasePath
The maximum payload size in bytes on incoming server requests
servermaxPayloadBytes 1048576
The Elasticsearch instance to use for all your queries
elasticsearchurl httplocalhost9200
preserve_elasticsearch_host true will send the hostname specified in `elasticsearch` If you set it to false
then the host you use to connect to this Kibana instance will be sent
elasticsearchpreserveHost true
Kibana uses an index in Elasticsearch to store saved searches visualizations
and dashboards It will create a new index if it doesnt already exist
kibanaindex kibana
The default application to load
kibanadefaultAppId discover
If your Elasticsearch is protected with basic auth these are the user credentials
used by the Kibana server to perform maintenance on the kibana_index at startup Your
Kibana users will still need to authenticate with Elasticsearch (which is proxied through
the Kibana server)
Page of 16 71 Page of 16 71
elasticsearchusername user
elasticsearchpassword pass
SSL for outgoing requests from the Kibana Server to the browser (PEM formatted)
serversslcert pathtoyourservercrt
serversslkey pathtoyourserverkey
Optional setting to validate that your Elasticsearch backend uses the same key files (PEM formatted)
elasticsearchsslcert pathtoyourclientcrt
elasticsearchsslkey pathtoyourclientkey
If you need to provide a CA certificate for your Elasticsearch instance put
the path of the pem file here
elasticsearchsslca pathtoyourCApem
Set to false to have a complete disregard for the validity of the SSL
certificate
elasticsearchsslverify true
Time in milliseconds to wait for elasticsearch to respond to pings defaults to
request_timeout setting
elasticsearchpingTimeout 1500
Time in milliseconds to wait for responses from the back end or elasticsearch
This must be gt 0
elasticsearchrequestTimeout 30000
Time in milliseconds for Elasticsearch to wait for responses from shards
Set to 0 to disable
elasticsearchshardTimeout 0
Time in milliseconds to wait for Elasticsearch at Kibana startup before retrying
elasticsearchstartupTimeout 5000
elasticsearchstartupTimeout 30000
Page of 17 71 Page of 17 71
Set the path to where you would like the process id file to be created
pidfile varrunkibanapid
If you would like to send the log output to a file you can set the path below
loggingdest stdout
Set this to true to suppress all logging output
loggingsilent false
Set this to true to suppress all logging output except for error messages
loggingquiet false
Set this to true to log all events including system usage information and all requests
loggingverbose false
Kibana13 is13 now13 installed13
Start13 Kibana13 To13 start13 Kibana13 go13 to13 the13 bin13 directory13 and13 run13 the13 kibana13 executable13
[rootelkforcase]$
[rootelkforcase]$ cd homekibanakibana-451-linux-x64bin
[rootelkforcase]$ kibana amp
[1] 22283
[rootelkforcase] log [095900542] [info][status][pluginkibana] Status changed from uninitialized to green - Ready
log [095900600] [info][status][pluginelasticsearch] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [095900689] [info][status][pluginkbn_vislib_vis_types] Status changed from uninitialized to green - Ready
log [095900717] [info][status][pluginmarkdown_vis] Status changed from uninitialized to green - Ready
log [095900721] [info][status][pluginmetric_vis] Status changed from uninitialized to green - Ready
log [095900814] [info][status][pluginspyModes] Status changed from uninitialized to green - Ready
log [095900934] [info][status][pluginstatusPage] Status changed from uninitialized to green - Ready
Page of 18 71 Page of 18 71
log [095900941] [info][status][plugintable_vis] Status changed from uninitialized to green - Ready
log [095901067] [info][listening] Server running at http00005601
log [095906083] [info][status][pluginelasticsearch] Status changed from yellow to yellow - No existing Kibana index found
log [095908942] [info][status][pluginelasticsearch] Status changed from yellow to green - Kibana index ready
Kibana13 is13 now13 started13
Broadcast13 Bluemix13 app13 log13 records13 Bluemix13 apps13 (which13 are13 shown13 in13 the13 US13 South13 Bluemix13 region13 of13 the13 System13 Architecture13 diagram)13 will13 produce13 logs13 in13 a13 syslog13 rfc542413 format13 using13 a13 Bluemix13 tool13 called13 Loggregator13
For13 Logstash13 to13 receive13 these13 logs13 each13 Bluemix13 app13 needs13 to13 broadcast13 these13 logs13 To13 do13 that13 run13 the13 following13 commands13 on13 each13 Bluemix13 app13 using13 the13 public13 IP13 address13 of13 the13 ELK13 virtual13 server13 the13 ELK13 tag13 to13 trace13 the13 process13 and13 insert13 the13 name13 of13 each13 app13 in13 turn13
cf cups elk -l syslogltpublicIPaddress5000 cf bind-service ltmyappgt elk
cf restage ltmyappgt
Install13 Logstash13 To13 install13 Logstash13 as13 root13 user13 create13 a13 directory13 for13 the13 installaon13 and13 untar13 the13 installaon13 file13 there13
13 13 [rootelkforcase]$ cd home
[rootelkforcase]$ mkdir logstash
[rootelkforcase]$ cp homeibmcloudlogstash-233targz homelogstash
[rootelkforcase]$ cd logstash
[rootelkforcase]$ gunzip logstash-233targz
[rootelkforcase]$ untar logstash-233tar
Write13 Logstash13 configura0on13 Logstash13 requires13 a13 configuraon13 file13 to13 be13 wricen13 to13 receive13 transform13 and13 output13 the13 log13 records13 to13 Elascsearch13 This13 secon13 explains13 the13 basic13 requirements13 of13 processing13 Bluemix13 app13 log13 records13 into13 Elascsearch13 (more13 complex13 configuraons13 are13 detailed13 later13 in13 this13 document13 for13 the13 integraons13 with13 third-shy‐party13 tools)13
Create13 a13 conf13 file13 as13 root13 user13
[rootelkforcase]$ cd homelogstashlogstash-233
[rootelkforcase]$ mkdir config
Page of 19 71 Page of 19 71
[rootelkforcase]$ cd config
[rootelkforcase]$ touch loggregatorconf
Edit13 the13 loggregator13 file13 with13 the13 following13 contents13
13 Input13 secon13 defines13 which13 log13 records13 to13 receive13
13 port13 and13 type13 reflect13 method13 selected13 for13 Cloud13 Foundry13 to13 broadcast13 the13 records13
input
tcp
port =gt 5000
type =gt syslog
13 Filter13 secon13 ndash13 transforms13 the13 data13
filter
Add a field to identify the records as Cloud Foundry based
mutate
add_field =gt format =gt cf
13 Output13 secon13 ndash13 sends13 the13 data13 to13 its13 desnaon(s)13
output
13 Use13 the13 elascsearch13 plugin13 to13 send13 the13 data13 to13 the13 elascsearch13 instance13
13 elasticsearch
hosts =gt localhost
stdout codec =gt rubydebug
13 End13 of13 Config13
Save13 the13 file13 Verify13 the13 format13 of13 the13 file13 is13 acceptable13 by13 running13 the13 following13 commands13
[rootelkforcase]$ cd homelogstashlogstash-233
[rootelkforcase]$ binlogstash agent --configtest -f configloggregatorconf
Page of 20 71 Page of 20 71
Configuraon13 is13 OK13
Start13 Logstash13 Aaer13 you13 have13 wricen13 and13 verified13 the13 Logstash13 configuraon13 you13 can13 start13 Logstash13 to13 process13 records13 from13 the13 Bluemix13 apps13 (via13 Cloud13 Foundry)13 Run13 the13 following13 as13 root13 user13
[rootelkforcase]$ cd homelogstashlogstash-233
[rootelkforcase]$ binlogstash agent -f configloggregatorconf
Logstash13 is13 now13 started13
Monitor13 Logstash13 processing13 Once13 Logstash13 is13 running13 and13 log13 records13 are13 being13 produced13 by13 the13 Bluemix13 apps13 you13 can13 monitor13 progress13 in13 the13 same13 window13 that13 was13 used13 to13 start13 Logstash13 Output13 is13 similar13 to13 the13 following13
13
The13 log13 records13 being13 broadcast13 from13 the13 Bluemix13 apps13 are13 now13 sent13 to13 Elascsearch13 from13 the13 Logstash13 output13 configuraon13 Records13 are13 sent13 in13 real13 me13 If13 Logstash13 is13 not13 running13 while13 log13 records13 are13 being13 produced13 then13 these13 records13 will13 not13 be13 processed13 by13 Logstash13 and13 will13 not13 be13 stored13 in13 Elascsearch13
Page of 21 71 Page of 21 71
Use13 ELK13 The13 log13 records13 from13 the13 Bluemix13 apps13 are13 now13 being13 stored13 and13 indexed13 by13 the13 Elascsearch13 component13 This13 data13 can13 be13 accessed13 via13 the13 Kibana13 applicaon13
Kibana13 Discover13 page13 Using13 a13 web13 browser13 either13 directly13 from13 a13 client13 or13 using13 the13 VNC13 client13 access13 Kibana13 using13 the13 publicIPaddress13 of13 the13 ELK13 virtual13 server13
hcpltpublicIPaddressgt560113
No13 userpassword13 credenals13 are13 required13
Kibana13 defaults13 to13 a13 search13 of13 all13 records13 ()13 in13 the13 last13 1513 minutes13 and13 returns13 the13 results13
13
More13 complex13 searches13 are13 possible13 by13 changing13 the13 me13 filter13 and13 supplying13 search13 queries13
Change13 the13 0me13 filter13 To13 change13 the13 me13 ranges13 click13 the13 clock13 symbol13 in13 the13 top-shy‐right13 of13 the13 Kibana13 Discover13 page13
13
You13 will13 see13 the13 following13 opons13
13
Page of 22 71 Page of 22 71
A13 variety13 of13 quick13 me13 ranges13 can13 be13 selected13 or13 you13 can13 switch13 to13 relave13 or13 absolute13 me13 ranges13
Quick13 me13 ranges13 allow13 switching13 between13 last13 X13 minutes13 to13 last13 Y13 hours13 or13 Z13 days13 etc13
Relave13 me13 ranges13 allow13 more13 specific13 versions13 of13 the13 Quick13 opons13 from13 specific13 minuteshoursdays13 to13 now13
13
Absolute13 me13 ranges13 specify13 the13 start13 and13 end13 mes13 for13 the13 searches13
13
Custom13 search13 queries13 You13 can13 use13 the13 text13 window13 of13 the13 Discover13 page13 to13 search13 for13 matches13 within13 the13 log13 records13 You13 can13 use13 Apache13 Lucene13 or13 Elascsearch13 query13 DSL13 syntax13
13
Page of 23 71 Page of 23 71
Saved13 searches13 You13 can13 save13 any13 custom13 search13 syntax13 and13 choice13 of13 me13 range13 for13 future13 ease13 of13 use13 Any13 relave13 or13 absolute13 me13 frame13 is13 saved13 as13 part13 of13 the13 search13 13
To13 save13 a13 search13
1 Set13 up13 and13 run13 the13 required13 syntax13 and13 me13 frame13
13
2 Click13 the13 Save13 Search13 icon13
13
3 Type13 in13 a13 name13 for13 the13 saved13 search13
Page of 24 71 Page of 24 71
13
7 To13 delete13 or13 update13 saved13 searches13 click13 13 manage13 saved13 searches13
13
Use13 the13 edit13 icons13 to13 make13 any13 changes13
Page of 26 71 Page of 26 71
13
Build13 simple13 Kibana13 histograms13 Kibana13 allows13 you13 to13 build13 charts13 and13 dashboards13 based13 on13 the13 search13 capability13 offered13 by13 the13 Discover13 page13 Saved13 searches13 can13 be13 used13 as13 the13 basis13 for13 charts13 and13 dashboards13 built13 using13 those13 charts13 To13 build13 a13 simple13 histogram13
1 Select13 the13 Visualize13 page13
13
2 Select13 Ver0cal13 Bar13 Chart13
13
3 To13 use13 a13 saved13 search13 click13 From13 a13 saved13 search13
Page of 27 71 Page of 27 71
13
4 Select13 a13 previously13 saved13 search13
13
5 Select13 to13 edit13 the13 x-shy‐axis
6 Edit13 the13 fields13 with13 informaon13 about13 the13 aggregaon13 field13 interval13 and13 custom13 label13
Page of 28 71 Page of 28 71
13
Page of 29 71 Page of 29 71
7 Click13 the13 green13 arrow13 The13 histogram13 is13 created13 and13 run13 based13 on13 the13 saved13 search
13
8 To13 save13 the13 histogram13 click13 the13 Save13 Visualiza0on13 icon13 and13 add13 a13 tle13
13
13
9 Click13 Save13
10 To13 view13 the13 visualisaon13 click13 the13 Dashboard13 page13
Page of 30 71 Page of 30 71
13
11 Click13 the13 Add13 Visualiza0on13 icon13
13
12 Select13 the13 visiualizaon13 that13 you13 just13 created13
13
Page of 31 71 Page of 31 71
13
13 To13 save13 the13 dashboard13 click13 the13 Save13 Dashboard13 icon13
Page of 32 71 Page of 32 71
13
14 Add13 a13 tle13 and13 select13 whether13 to13 store13 the13 selected13 me13 frame13 with13 the13 dashboard13
13
15 Click13 Save13
16 You13 can13 edit13 the13 shape13 and13 number13 of13 charts13 and13 save13 the13 dashboard13
17 To13 re-shy‐run13 the13 dashboard13 select13 the13 Dashboard13 page13 and13 click13 the13 Load13 Saved13 Dashboard13 icon13
Page of 33 71 Page of 33 71
13
18 Select13 the13 saved13 dashboard13 from13 the13 list13
13
19 The13 dashboard13 runs13
13
Page of 34 71 Page of 34 71
Page of 35 71 Page of 35 71
Aler0ng13 with13 ELK13 ELK13 examines13 the13 log13 records13 in13 process13 to13 send13 alerts13 to13 third-shy‐party13 alerng13 event13 management13 and13 social13 media13 applicaons13 As13 shown13 previously13 in13 the13 ELK13 architecture13 the13 ELK13 strategy13 for13 alerng13 is13 to13 send13 alerts13 from13 the13 Logstash13 component13 directly13 Therefore13 log13 records13 and13 alerts13 are13 sent13 to13 both13 Elascsearch13 and13 third-shy‐party13 tools13 at13 the13 same13 instant13 using13 the13 output13 secon13 of13 the13 Logstash13 configuraon13 Any13 alerts13 are13 generated13 during13 the13 filter13 secon13 processing13 of13 the13 Logstash13 configuraon13
Create13 alerts13 in13 Logstash13 The13 Logstash13 filter13 secon13 is13 used13 to13 trigger13 alerts13 based13 on13 the13 log13 records13 being13 processed13 For13 example13 log13 records13 can13 be13 checked13 for13 errors13 warnings13 crashed13 or13 stopped13 messages13 and13 generate13 an13 alert13
Here13 is13 an13 example13 Logstash13 filter13 code13
13 if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
The13 specific13 tags13 are13 set13 against13 the13 log13 records13 to13 trigger13 the13 alert13 later13 in13 the13 configuraon13 in13 the13 output13 secon13
Control13 the13 number13 of13 generated13 alerts13 using13 the13 Logstash13 throcle13 plugin13 so13 that13 either13 a13 lot13 of13 instances13 in13 a13 fixed13 me13 period13 are13 required13 before13 an13 alert13 is13 sent13 or13 a13 specific13 number13 of13 instances13 triggers13 the13 alert13
Page of 36 71 Page of 36 71
Edit13 the13 plugin13 parameters13 to13 meet13 system13 requirements13
13 13 13 13 13 if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 10
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle13
after_count =gt 20
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 30
period =gt 1200
key =gt servicename_alarm_warning
Page of 37 71 Page of 37 71
add_tag =gt servicename_throttled
13 13 13 13 13 13 13 13 13
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 40
period =gt 1500
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
Enrich13 alerts13 with13 message13 detail13 Some13 of13 the13 third-shy‐party13 integraons13 benefit13 from13 having13 components13 of13 the13 raw13 log13 record13 sent13 in13 addion13 to13 the13 raw13 record13 or13 fields13 in13 the13 record13 edited13 transformed13 or13 translated13
Grok13 Logstash13 allows13 the13 log13 record13 to13 be13 broken13 up13 into13 logical13 secons13 and13 each13 secon13 stored13 in13 a13 new13 parameter13 using13 the13 grok13 parameter13 in13 the13 filter13 secon13 Here13 is13 an13 example13 grok13
13 grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
To13 build13 the13 grok13 statement13 Logstash13 offers13 an13 online13 tool13 at13 hcpsgrokdebugherokuappcom13
Translate13 If13 specific13 fields13 in13 the13 log13 record13 are13 coded13 (for13 example13 the13 Bluemix13 app13 name13 is13 oaen13 not13 sent13 in13 the13 syslog13 messages13 instead13 the13 GUID13 is13 sent13 which13 is13 not13 useful)13 then13 the13 filter13 secon13 translate13 plugin13 can13 be13 used13 to13 add13 new13 fields13 by13 holding13 the13 decoded13 versions13 of13 those13 values13
13 translate
field =gt data1
destination =gt data1
override =gt true
Page of 38 71 Page of 38 71
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
dc50f6e4-c052-436a-bf27-95ba392c4228 monverifyservicemanagement
0c677e82-5bfc-4e38-9155-e416249e3ee6 ms-catalog-postpyretic-pomiculture
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
However13 the13 translate13 plugin13 is13 not13 included13 with13 the13 Logstash13 installaon13 In13 the13 next13 secon13 I13 will13 13 describe13 how13 to13 add13 new13 plugins13 to13 the13 Logstash13 instance13
Add13 custom13 plugins13 to13 Logstash13 As13 an13 open13 source13 tool13 Logstash13 encourages13 users13 to13 develop13 content13 for13 all13 users13 to13 share13 Some13 of13 the13 plugins13 that13 were13 created13 by13 our13 community13 include13 a13 translaon13 plugin13 an13 output13 Slack13 plugin13 and13 an13 output13 syslog13 plugin13 Letrsquos13 look13 at13 how13 to13 add13 these13 plugins13 to13 your13 environment13
Filter13 translate13 plugin13 To13 add13 the13 translate13 plugin13
1 Download13 the13 plugin13
2 Copy13 the13 plugin13 file13 logstash-shy‐filter-shy‐translate-shy‐214gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-filter-translate 214
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Output13 Slack13 plugin13 To13 add13 the13 Slack13 plugin13
Page of 39 71 Page of 39 71
1 Download13 the13 plugin13 13
2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐slack-shy‐014gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-output-slack 014
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Output13 syslog13 plugin13 To13 add13 the13 syslog13 plugin13
1 Download13 the13 plugin13 13
2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐syslog-shy‐214gem13 to13 the13 ELK13 Virtual13 Server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-output-syslog 214
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Integrate13 Logstash13 with13 Slack13 To13 send13 generated13 alerts13 to13 Slack13 complete13 these13 steps13
1 Create13 a13 Slack13 integraon13 13
2 Use13 the13 Slack13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 Slack13 integraon13
113 Login13 to13 Slack13 and13 go13 to13 the13 Apps13 page13 13
Page of 40 71 Page of 40 71
13
313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13
13
A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13
13
Page of 42 71 Page of 42 71
The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13
13
The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13
13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
slack
url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5
channel =gt cloud-operations-mon
username =gt nick_cawood
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13
Page of 43 71 Page of 43 71
Page of 44 71 Page of 44 71
In13 Slack13 the13 alerts13 look13 like13 this13
13
Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13
1 Create13 a13 PagerDuty13 integraon13 13
2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13
1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13
13
2 Click13 Add13 New13 Service13
13
313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13
Page of 45 71 Page of 45 71
13
413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13
513 Click13 Add13 Service13
13
The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13
Page of 46 71 Page of 46 71
13
The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt be7eb1ea9c224018923e818d3e891d0b
incident_key =gt logstashhosttype
Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13
Page of 47 71 Page of 47 71
13
Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13
1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13
2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13
3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13
4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13
13 13 http
url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo
http_method =gt post
format =gt json
For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13
13 13 syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
Page of 48 71 Page of 48 71
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13
13
The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13
if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2
This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13
Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13
Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13
Logstash13 severity Shown13 in13 Omnibus13 as
alert Indeterminate
crical Warning
debug 7
emergency Clear
error Minor
informaonal 6
noce Crical
warning Major
Page of 49 71 Page of 49 71
13
Page of 50 71 Page of 50 71
Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13
input
tcp
port =gt 5000
type =gt syslog
filter
mutate
add_field =gt format =gt cf
grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
translate
field =gt data1
destination =gt data1
override =gt true
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
=== Check Alarm Criteria ===
Standard Key Words
if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
=== Alarm Exclusions ===
if [message] =~ (i)sometexthere
mutate remove_tag =gt servicename_alarm
Throttle alarms so we do not hit PD rate limits
Page of 52 71 Page of 52 71
if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 5
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle
after_count =gt 5
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 5
period =gt 1200
key =gt servicename_alarm_warning
add_tag =gt servicename_throttled
Page of 53 71 Page of 53 71
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
output
elasticsearch
hosts =gt localhost
stdout codec =gt rubydebug
if servicename_alarm_err in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
Page of 54 71 Page of 54 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 55 71 Page of 55 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
Page of 56 71 Page of 56 71
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert err message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_error in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
Page of 57 71 Page of 57 71
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 58 71 Page of 58 71
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 59 71 Page of 59 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert error message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
Page of 60 71 Page of 60 71
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
Page of 61 71 Page of 61 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 62 71 Page of 62 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt x
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert fatal message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
Page of 63 71 Page of 63 71
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
Page of 64 71 Page of 64 71
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 65 71 Page of 65 71
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt heavy_exclamation_mark
Page of 66 71 Page of 66 71
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert WARNING message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 67 71 Page of 67 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
Page of 68 71 Page of 68 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 69 71 Page of 69 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
Page of 70 71 Page of 70 71
Page of 71 71 Page of 71 71
Copy13 files13 to13 the13 virtual13 server13 To13 move13 installaon13 files13 to13 the13 server13 go13 to13 the13 UNIX13 or13 Linux13 window13 you13 used13 to13 create13 the13 SSH13 key13 and13 the13 directory13 where13 the13 ltuniquekeynamegt13 files13 exist13 Move13 the13 installaon13 files13 to13 the13 same13 locaon13
Run13 the13 following13 command13 referencing13 the13 SSH13 keyname13 the13 public13 IP13 address13 Bluemix13 allocated13 for13 the13 virtual13 server13 and13 the13 password13 for13 the13 SSH13 key13
[userserver ~]$ scp ndashi ltuniquekeynamegt ltINSTALLATIONFILENAMEgt ibmcloudltpublicIPaddressgthomeibmcloud
Enter13 the13 passphrase13 for13 key13 cloudkey313
This13 copies13 the13 file13 using13 the13 ibmcloud13 user13 to13 the13 homeibmcloud13 directory13 on13 the13 virtual13 server13
Install13 and13 access13 the13 virtual13 server13 via13 VNC13 server13 Using13 the13 ssh13 command13 to13 access13 the13 virtual13 server13 is13 not13 the13 most13 stable13 method13 and13 does13 not13 offer13 X-shy‐Windows13 support13 Therefore13 we13 suggest13 that13 you13 use13 a13 VNC13 client13 to13 access13 the13 virtual13 server13
Installing13 a13 VNC13 client13 should13 be13 done13 as13 the13 root13 user13 Use13 the13 yum13 command13 as13 follows13
[ibmcloudelkforcase ~]$ yum groupinstall GNOME Desktop
[ibmcloudelkforcase ~]$ yum install tigervnc-server
These13 commands13 first13 install13 a13 desktop13 style13 OS13 user13 interface13 and13 then13 a13 VNC13 client13 to13 remotely13 access13 the13 desktop13 style13 OS13
Then13 as13 either13 the13 root13 user13 or13 another13 user13 (example13 below13 uses13 elk)13 run13 the13 following13 command13 to13 create13 the13 VNC13 client13 connecon13
[elkelkforcase ~]$ vncserver
New elkforcase2 (elk) desktop is elkforcase2
Starting applications specified in homeelkvncxstartup
Log file is homeelkvncelkforcase2log
[elkelkforcase ~]$
Install13 a13 VNC13 client13 on13 the13 client13 machine13 used13 to13 access13 the13 Internet13 and13 (in13 the13 example13 above)13 use13 the13 command13 ltpublicIPadddressgt213 and13 the13 password13 for13 ELK13 to13 access13 the13 virtual13 server13
Using13 a13 VNC13 client13 means13 that13 you13 can13 leave13 long13 duraon13 installaons13 running13 without13 fear13 of13 the13 terminal13 losing13 connecon13 and13 the13 installaons13 failing13
Install13 the13 ELK13 stack13 Download13 the13 source13 soaware13 code13 for13 the13 ELK13 stack13 from13 the13 ELK13 website13 13
At13 the13 me13 of13 wring13 the13 downloaded13 soaware13 files13 were13
Page of 13 71 Page of 13 71
13
Transfer13 the13 files13 to13 the13 virtual13 server13 (using13 the13 scp13 method13 detailed13 above)13
Install13 Elas0csearch13 As13 a13 root13 user13 create13 a13 non-shy‐root13 user13 with13 the13 adduser13 command13
Create13 a13 directory13 to13 install13 Elascsearch13 into13 and13 untar13 the13 contents13 of13 the13 Elascsearch13 install13 file13 into13 that13 directory13
[elkelkforcase]$ cd home
[elkelkforcase]$ mkdir elasticsearch
[elkelkforcase]$ cp homeibmcloudelasticsearch-233targz homeelasticsearch
[elkelkforcase]$ cd elasticsearch
[elkelkforcase]$ gunzip elasticsearch-233targz
[elkelkforcase]$ untar elasticsearch-233tar
Elascsearch13 is13 now13 installed13
Start13 Elas0csearch13 To13 start13 Elascsearch13 go13 to13 the13 bin13 directory13 as13 the13 non-shy‐root13 user13 when13 you13 installed13 it13 and13 run13 the13 elascsearch13 executable13
[elkelkforcase]$ cd homeelasticsearchelasticsearch-233bin
[elkelkforcase]$ elasticsearch amp
[2] 22211
[elkelkforcase]$ [2016-07-05 133857730][INFO ][node] [Arsenal] version[233] pid[22211] build[218bdf12016-05-17T154004Z]
[2016-07-05 133857740][INFO ][node] [Arsenal] initializing
[2016-07-05 133858984][INFO ][plugins] [Arsenal] modules [reindex lang-expression lang-groovy] plugins [] sites []
[2016-07-05 133859146][INFO ][env] [Arsenal] using [1] data paths mounts [[ (rootfs)]] net usable_space [155gb] net total_space [199gb] spins [unknown] types [rootfs]
[2016-07-05 133859146][INFO ][env] [Arsenal] heap size [10156mb] compressed ordinary object pointers [true]
Page of 14 71 Page of 14 71
[2016-07-05 133859147][WARN ][env] [Arsenal] max file descriptors [4096] for elasticsearch process likely too low consider increasing to at least [65536]
[2016-07-05 133903526][INFO ][node] [Arsenal] initialized
[2016-07-05 133903526][INFO ][node] [Arsenal] starting
To13 test13 that13 Elascsearch13 is13 running13 correctly13 open13 the13 browser13 in13 the13 VNC13 client13 and13 run13 localhost920013
13
Install13 Kibana13 ELK13 recommends13 installing13 and13 running13 Kibana13 as13 root13 13
1 As13 root13 user13 create13 an13 installaon13 directory13 and13 untar13 the13 installaon13 file13
[rootelkforcase]$ cd home
[rootelkforcase]$ mkdir kibana
[rootelkforcase]$ cp homeibmcloudkibana-451-linux-x64targz homekibana
[rootelkforcase]$ cd kibana
[rootelkforcase]$ gunzip kibana-451-linux-x64targz
[rootelkforcase]$ untar kibana-451-linux-x64tar
2 Modify13 the13 manifestyml13 configuraon13 file13 Go13 to13 the13 kibana-shy‐451-shy‐linux-shy‐x6413 directory13
[rootelkforcase]$ cd homekibanakibana-451-linux-x64
3 Edit13 the13 manifestyml13 file13 so13 the13 contents13 are13
applicaons13
-shy‐ name13 kibana-shy‐in-shy‐bluemix13
memory13 128M13
host13 kibana-shy‐in-shy‐bluemix13
buildpack13 hcpsgithubcomcloudfoundry-shy‐communitynginx-shy‐buildpackgit13
Page of 15 71 Page of 15 71
4 Edit13 the13 configkibanayml13 file13 so13 the13 contents13 are13
Kibana is served by a back end server This controls which port to use
port 5601
The host to bind the server to
host 0000
If you are running kibana behind a proxy and want to mount it at a path
specify that path here The basePath cant end in a slash
serverbasePath
The maximum payload size in bytes on incoming server requests
servermaxPayloadBytes 1048576
The Elasticsearch instance to use for all your queries
elasticsearchurl httplocalhost9200
preserve_elasticsearch_host true will send the hostname specified in `elasticsearch` If you set it to false
then the host you use to connect to this Kibana instance will be sent
elasticsearchpreserveHost true
Kibana uses an index in Elasticsearch to store saved searches visualizations
and dashboards It will create a new index if it doesnt already exist
kibanaindex kibana
The default application to load
kibanadefaultAppId discover
If your Elasticsearch is protected with basic auth these are the user credentials
used by the Kibana server to perform maintenance on the kibana_index at startup Your
Kibana users will still need to authenticate with Elasticsearch (which is proxied through
the Kibana server)
Page of 16 71 Page of 16 71
elasticsearchusername user
elasticsearchpassword pass
SSL for outgoing requests from the Kibana Server to the browser (PEM formatted)
serversslcert pathtoyourservercrt
serversslkey pathtoyourserverkey
Optional setting to validate that your Elasticsearch backend uses the same key files (PEM formatted)
elasticsearchsslcert pathtoyourclientcrt
elasticsearchsslkey pathtoyourclientkey
If you need to provide a CA certificate for your Elasticsearch instance put
the path of the pem file here
elasticsearchsslca pathtoyourCApem
Set to false to have a complete disregard for the validity of the SSL
certificate
elasticsearchsslverify true
Time in milliseconds to wait for elasticsearch to respond to pings defaults to
request_timeout setting
elasticsearchpingTimeout 1500
Time in milliseconds to wait for responses from the back end or elasticsearch
This must be gt 0
elasticsearchrequestTimeout 30000
Time in milliseconds for Elasticsearch to wait for responses from shards
Set to 0 to disable
elasticsearchshardTimeout 0
Time in milliseconds to wait for Elasticsearch at Kibana startup before retrying
elasticsearchstartupTimeout 5000
elasticsearchstartupTimeout 30000
Page of 17 71 Page of 17 71
Set the path to where you would like the process id file to be created
pidfile varrunkibanapid
If you would like to send the log output to a file you can set the path below
loggingdest stdout
Set this to true to suppress all logging output
loggingsilent false
Set this to true to suppress all logging output except for error messages
loggingquiet false
Set this to true to log all events including system usage information and all requests
loggingverbose false
Kibana13 is13 now13 installed13
Start13 Kibana13 To13 start13 Kibana13 go13 to13 the13 bin13 directory13 and13 run13 the13 kibana13 executable13
[rootelkforcase]$
[rootelkforcase]$ cd homekibanakibana-451-linux-x64bin
[rootelkforcase]$ kibana amp
[1] 22283
[rootelkforcase] log [095900542] [info][status][pluginkibana] Status changed from uninitialized to green - Ready
log [095900600] [info][status][pluginelasticsearch] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [095900689] [info][status][pluginkbn_vislib_vis_types] Status changed from uninitialized to green - Ready
log [095900717] [info][status][pluginmarkdown_vis] Status changed from uninitialized to green - Ready
log [095900721] [info][status][pluginmetric_vis] Status changed from uninitialized to green - Ready
log [095900814] [info][status][pluginspyModes] Status changed from uninitialized to green - Ready
log [095900934] [info][status][pluginstatusPage] Status changed from uninitialized to green - Ready
Page of 18 71 Page of 18 71
log [095900941] [info][status][plugintable_vis] Status changed from uninitialized to green - Ready
log [095901067] [info][listening] Server running at http00005601
log [095906083] [info][status][pluginelasticsearch] Status changed from yellow to yellow - No existing Kibana index found
log [095908942] [info][status][pluginelasticsearch] Status changed from yellow to green - Kibana index ready
Kibana13 is13 now13 started13
Broadcast13 Bluemix13 app13 log13 records13 Bluemix13 apps13 (which13 are13 shown13 in13 the13 US13 South13 Bluemix13 region13 of13 the13 System13 Architecture13 diagram)13 will13 produce13 logs13 in13 a13 syslog13 rfc542413 format13 using13 a13 Bluemix13 tool13 called13 Loggregator13
For13 Logstash13 to13 receive13 these13 logs13 each13 Bluemix13 app13 needs13 to13 broadcast13 these13 logs13 To13 do13 that13 run13 the13 following13 commands13 on13 each13 Bluemix13 app13 using13 the13 public13 IP13 address13 of13 the13 ELK13 virtual13 server13 the13 ELK13 tag13 to13 trace13 the13 process13 and13 insert13 the13 name13 of13 each13 app13 in13 turn13
cf cups elk -l syslogltpublicIPaddress5000 cf bind-service ltmyappgt elk
cf restage ltmyappgt
Install13 Logstash13 To13 install13 Logstash13 as13 root13 user13 create13 a13 directory13 for13 the13 installaon13 and13 untar13 the13 installaon13 file13 there13
13 13 [rootelkforcase]$ cd home
[rootelkforcase]$ mkdir logstash
[rootelkforcase]$ cp homeibmcloudlogstash-233targz homelogstash
[rootelkforcase]$ cd logstash
[rootelkforcase]$ gunzip logstash-233targz
[rootelkforcase]$ untar logstash-233tar
Write13 Logstash13 configura0on13 Logstash13 requires13 a13 configuraon13 file13 to13 be13 wricen13 to13 receive13 transform13 and13 output13 the13 log13 records13 to13 Elascsearch13 This13 secon13 explains13 the13 basic13 requirements13 of13 processing13 Bluemix13 app13 log13 records13 into13 Elascsearch13 (more13 complex13 configuraons13 are13 detailed13 later13 in13 this13 document13 for13 the13 integraons13 with13 third-shy‐party13 tools)13
Create13 a13 conf13 file13 as13 root13 user13
[rootelkforcase]$ cd homelogstashlogstash-233
[rootelkforcase]$ mkdir config
Page of 19 71 Page of 19 71
[rootelkforcase]$ cd config
[rootelkforcase]$ touch loggregatorconf
Edit13 the13 loggregator13 file13 with13 the13 following13 contents13
13 Input13 secon13 defines13 which13 log13 records13 to13 receive13
13 port13 and13 type13 reflect13 method13 selected13 for13 Cloud13 Foundry13 to13 broadcast13 the13 records13
input
tcp
port =gt 5000
type =gt syslog
13 Filter13 secon13 ndash13 transforms13 the13 data13
filter
Add a field to identify the records as Cloud Foundry based
mutate
add_field =gt format =gt cf
13 Output13 secon13 ndash13 sends13 the13 data13 to13 its13 desnaon(s)13
output
13 Use13 the13 elascsearch13 plugin13 to13 send13 the13 data13 to13 the13 elascsearch13 instance13
13 elasticsearch
hosts =gt localhost
stdout codec =gt rubydebug
13 End13 of13 Config13
Save13 the13 file13 Verify13 the13 format13 of13 the13 file13 is13 acceptable13 by13 running13 the13 following13 commands13
[rootelkforcase]$ cd homelogstashlogstash-233
[rootelkforcase]$ binlogstash agent --configtest -f configloggregatorconf
Page of 20 71 Page of 20 71
Configuraon13 is13 OK13
Start13 Logstash13 Aaer13 you13 have13 wricen13 and13 verified13 the13 Logstash13 configuraon13 you13 can13 start13 Logstash13 to13 process13 records13 from13 the13 Bluemix13 apps13 (via13 Cloud13 Foundry)13 Run13 the13 following13 as13 root13 user13
[rootelkforcase]$ cd homelogstashlogstash-233
[rootelkforcase]$ binlogstash agent -f configloggregatorconf
Logstash13 is13 now13 started13
Monitor13 Logstash13 processing13 Once13 Logstash13 is13 running13 and13 log13 records13 are13 being13 produced13 by13 the13 Bluemix13 apps13 you13 can13 monitor13 progress13 in13 the13 same13 window13 that13 was13 used13 to13 start13 Logstash13 Output13 is13 similar13 to13 the13 following13
13
The13 log13 records13 being13 broadcast13 from13 the13 Bluemix13 apps13 are13 now13 sent13 to13 Elascsearch13 from13 the13 Logstash13 output13 configuraon13 Records13 are13 sent13 in13 real13 me13 If13 Logstash13 is13 not13 running13 while13 log13 records13 are13 being13 produced13 then13 these13 records13 will13 not13 be13 processed13 by13 Logstash13 and13 will13 not13 be13 stored13 in13 Elascsearch13
Page of 21 71 Page of 21 71
Use13 ELK13 The13 log13 records13 from13 the13 Bluemix13 apps13 are13 now13 being13 stored13 and13 indexed13 by13 the13 Elascsearch13 component13 This13 data13 can13 be13 accessed13 via13 the13 Kibana13 applicaon13
Kibana13 Discover13 page13 Using13 a13 web13 browser13 either13 directly13 from13 a13 client13 or13 using13 the13 VNC13 client13 access13 Kibana13 using13 the13 publicIPaddress13 of13 the13 ELK13 virtual13 server13
hcpltpublicIPaddressgt560113
No13 userpassword13 credenals13 are13 required13
Kibana13 defaults13 to13 a13 search13 of13 all13 records13 ()13 in13 the13 last13 1513 minutes13 and13 returns13 the13 results13
13
More13 complex13 searches13 are13 possible13 by13 changing13 the13 me13 filter13 and13 supplying13 search13 queries13
Change13 the13 0me13 filter13 To13 change13 the13 me13 ranges13 click13 the13 clock13 symbol13 in13 the13 top-shy‐right13 of13 the13 Kibana13 Discover13 page13
13
You13 will13 see13 the13 following13 opons13
13
Page of 22 71 Page of 22 71
A13 variety13 of13 quick13 me13 ranges13 can13 be13 selected13 or13 you13 can13 switch13 to13 relave13 or13 absolute13 me13 ranges13
Quick13 me13 ranges13 allow13 switching13 between13 last13 X13 minutes13 to13 last13 Y13 hours13 or13 Z13 days13 etc13
Relave13 me13 ranges13 allow13 more13 specific13 versions13 of13 the13 Quick13 opons13 from13 specific13 minuteshoursdays13 to13 now13
13
Absolute13 me13 ranges13 specify13 the13 start13 and13 end13 mes13 for13 the13 searches13
13
Custom13 search13 queries13 You13 can13 use13 the13 text13 window13 of13 the13 Discover13 page13 to13 search13 for13 matches13 within13 the13 log13 records13 You13 can13 use13 Apache13 Lucene13 or13 Elascsearch13 query13 DSL13 syntax13
13
Page of 23 71 Page of 23 71
Saved13 searches13 You13 can13 save13 any13 custom13 search13 syntax13 and13 choice13 of13 me13 range13 for13 future13 ease13 of13 use13 Any13 relave13 or13 absolute13 me13 frame13 is13 saved13 as13 part13 of13 the13 search13 13
To13 save13 a13 search13
1 Set13 up13 and13 run13 the13 required13 syntax13 and13 me13 frame13
13
2 Click13 the13 Save13 Search13 icon13
13
3 Type13 in13 a13 name13 for13 the13 saved13 search13
Page of 24 71 Page of 24 71
13
7 To13 delete13 or13 update13 saved13 searches13 click13 13 manage13 saved13 searches13
13
Use13 the13 edit13 icons13 to13 make13 any13 changes13
Page of 26 71 Page of 26 71
13
Build13 simple13 Kibana13 histograms13 Kibana13 allows13 you13 to13 build13 charts13 and13 dashboards13 based13 on13 the13 search13 capability13 offered13 by13 the13 Discover13 page13 Saved13 searches13 can13 be13 used13 as13 the13 basis13 for13 charts13 and13 dashboards13 built13 using13 those13 charts13 To13 build13 a13 simple13 histogram13
1 Select13 the13 Visualize13 page13
13
2 Select13 Ver0cal13 Bar13 Chart13
13
3 To13 use13 a13 saved13 search13 click13 From13 a13 saved13 search13
Page of 27 71 Page of 27 71
13
4 Select13 a13 previously13 saved13 search13
13
5 Select13 to13 edit13 the13 x-shy‐axis
6 Edit13 the13 fields13 with13 informaon13 about13 the13 aggregaon13 field13 interval13 and13 custom13 label13
Page of 28 71 Page of 28 71
13
Page of 29 71 Page of 29 71
7 Click13 the13 green13 arrow13 The13 histogram13 is13 created13 and13 run13 based13 on13 the13 saved13 search
13
8 To13 save13 the13 histogram13 click13 the13 Save13 Visualiza0on13 icon13 and13 add13 a13 tle13
13
13
9 Click13 Save13
10 To13 view13 the13 visualisaon13 click13 the13 Dashboard13 page13
Page of 30 71 Page of 30 71
13
11 Click13 the13 Add13 Visualiza0on13 icon13
13
12 Select13 the13 visiualizaon13 that13 you13 just13 created13
13
Page of 31 71 Page of 31 71
13
13 To13 save13 the13 dashboard13 click13 the13 Save13 Dashboard13 icon13
Page of 32 71 Page of 32 71
13
14 Add13 a13 tle13 and13 select13 whether13 to13 store13 the13 selected13 me13 frame13 with13 the13 dashboard13
13
15 Click13 Save13
16 You13 can13 edit13 the13 shape13 and13 number13 of13 charts13 and13 save13 the13 dashboard13
17 To13 re-shy‐run13 the13 dashboard13 select13 the13 Dashboard13 page13 and13 click13 the13 Load13 Saved13 Dashboard13 icon13
Page of 33 71 Page of 33 71
13
18 Select13 the13 saved13 dashboard13 from13 the13 list13
13
19 The13 dashboard13 runs13
13
Page of 34 71 Page of 34 71
Page of 35 71 Page of 35 71
Aler0ng13 with13 ELK13 ELK13 examines13 the13 log13 records13 in13 process13 to13 send13 alerts13 to13 third-shy‐party13 alerng13 event13 management13 and13 social13 media13 applicaons13 As13 shown13 previously13 in13 the13 ELK13 architecture13 the13 ELK13 strategy13 for13 alerng13 is13 to13 send13 alerts13 from13 the13 Logstash13 component13 directly13 Therefore13 log13 records13 and13 alerts13 are13 sent13 to13 both13 Elascsearch13 and13 third-shy‐party13 tools13 at13 the13 same13 instant13 using13 the13 output13 secon13 of13 the13 Logstash13 configuraon13 Any13 alerts13 are13 generated13 during13 the13 filter13 secon13 processing13 of13 the13 Logstash13 configuraon13
Create13 alerts13 in13 Logstash13 The13 Logstash13 filter13 secon13 is13 used13 to13 trigger13 alerts13 based13 on13 the13 log13 records13 being13 processed13 For13 example13 log13 records13 can13 be13 checked13 for13 errors13 warnings13 crashed13 or13 stopped13 messages13 and13 generate13 an13 alert13
Here13 is13 an13 example13 Logstash13 filter13 code13
13 if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
The13 specific13 tags13 are13 set13 against13 the13 log13 records13 to13 trigger13 the13 alert13 later13 in13 the13 configuraon13 in13 the13 output13 secon13
Control13 the13 number13 of13 generated13 alerts13 using13 the13 Logstash13 throcle13 plugin13 so13 that13 either13 a13 lot13 of13 instances13 in13 a13 fixed13 me13 period13 are13 required13 before13 an13 alert13 is13 sent13 or13 a13 specific13 number13 of13 instances13 triggers13 the13 alert13
Page of 36 71 Page of 36 71
Edit13 the13 plugin13 parameters13 to13 meet13 system13 requirements13
13 13 13 13 13 if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 10
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle13
after_count =gt 20
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 30
period =gt 1200
key =gt servicename_alarm_warning
Page of 37 71 Page of 37 71
add_tag =gt servicename_throttled
13 13 13 13 13 13 13 13 13
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 40
period =gt 1500
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
Enrich13 alerts13 with13 message13 detail13 Some13 of13 the13 third-shy‐party13 integraons13 benefit13 from13 having13 components13 of13 the13 raw13 log13 record13 sent13 in13 addion13 to13 the13 raw13 record13 or13 fields13 in13 the13 record13 edited13 transformed13 or13 translated13
Grok13 Logstash13 allows13 the13 log13 record13 to13 be13 broken13 up13 into13 logical13 secons13 and13 each13 secon13 stored13 in13 a13 new13 parameter13 using13 the13 grok13 parameter13 in13 the13 filter13 secon13 Here13 is13 an13 example13 grok13
13 grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
To13 build13 the13 grok13 statement13 Logstash13 offers13 an13 online13 tool13 at13 hcpsgrokdebugherokuappcom13
Translate13 If13 specific13 fields13 in13 the13 log13 record13 are13 coded13 (for13 example13 the13 Bluemix13 app13 name13 is13 oaen13 not13 sent13 in13 the13 syslog13 messages13 instead13 the13 GUID13 is13 sent13 which13 is13 not13 useful)13 then13 the13 filter13 secon13 translate13 plugin13 can13 be13 used13 to13 add13 new13 fields13 by13 holding13 the13 decoded13 versions13 of13 those13 values13
13 translate
field =gt data1
destination =gt data1
override =gt true
Page of 38 71 Page of 38 71
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
dc50f6e4-c052-436a-bf27-95ba392c4228 monverifyservicemanagement
0c677e82-5bfc-4e38-9155-e416249e3ee6 ms-catalog-postpyretic-pomiculture
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
However13 the13 translate13 plugin13 is13 not13 included13 with13 the13 Logstash13 installaon13 In13 the13 next13 secon13 I13 will13 13 describe13 how13 to13 add13 new13 plugins13 to13 the13 Logstash13 instance13
Add13 custom13 plugins13 to13 Logstash13 As13 an13 open13 source13 tool13 Logstash13 encourages13 users13 to13 develop13 content13 for13 all13 users13 to13 share13 Some13 of13 the13 plugins13 that13 were13 created13 by13 our13 community13 include13 a13 translaon13 plugin13 an13 output13 Slack13 plugin13 and13 an13 output13 syslog13 plugin13 Letrsquos13 look13 at13 how13 to13 add13 these13 plugins13 to13 your13 environment13
Filter13 translate13 plugin13 To13 add13 the13 translate13 plugin13
1 Download13 the13 plugin13
2 Copy13 the13 plugin13 file13 logstash-shy‐filter-shy‐translate-shy‐214gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-filter-translate 214
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Output13 Slack13 plugin13 To13 add13 the13 Slack13 plugin13
Page of 39 71 Page of 39 71
1 Download13 the13 plugin13 13
2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐slack-shy‐014gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-output-slack 014
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Output13 syslog13 plugin13 To13 add13 the13 syslog13 plugin13
1 Download13 the13 plugin13 13
2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐syslog-shy‐214gem13 to13 the13 ELK13 Virtual13 Server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-output-syslog 214
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Integrate13 Logstash13 with13 Slack13 To13 send13 generated13 alerts13 to13 Slack13 complete13 these13 steps13
1 Create13 a13 Slack13 integraon13 13
2 Use13 the13 Slack13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 Slack13 integraon13
113 Login13 to13 Slack13 and13 go13 to13 the13 Apps13 page13 13
Page of 40 71 Page of 40 71
13
313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13
13
A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13
13
Page of 42 71 Page of 42 71
The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13
13
The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13
13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
slack
url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5
channel =gt cloud-operations-mon
username =gt nick_cawood
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13
Page of 43 71 Page of 43 71
Page of 44 71 Page of 44 71
In13 Slack13 the13 alerts13 look13 like13 this13
13
Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13
1 Create13 a13 PagerDuty13 integraon13 13
2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13
1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13
13
2 Click13 Add13 New13 Service13
13
313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13
Page of 45 71 Page of 45 71
13
413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13
513 Click13 Add13 Service13
13
The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13
Page of 46 71 Page of 46 71
13
The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt be7eb1ea9c224018923e818d3e891d0b
incident_key =gt logstashhosttype
Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13
Page of 47 71 Page of 47 71
13
Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13
1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13
2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13
3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13
4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13
13 13 http
url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo
http_method =gt post
format =gt json
For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13
13 13 syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
Page of 48 71 Page of 48 71
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13
13
The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13
if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2
This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13
Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13
Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13
Logstash13 severity Shown13 in13 Omnibus13 as
alert Indeterminate
crical Warning
debug 7
emergency Clear
error Minor
informaonal 6
noce Crical
warning Major
Page of 49 71 Page of 49 71
13
Page of 50 71 Page of 50 71
Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13
input
tcp
port =gt 5000
type =gt syslog
filter
mutate
add_field =gt format =gt cf
grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
translate
field =gt data1
destination =gt data1
override =gt true
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
=== Check Alarm Criteria ===
Standard Key Words
if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
=== Alarm Exclusions ===
if [message] =~ (i)sometexthere
mutate remove_tag =gt servicename_alarm
Throttle alarms so we do not hit PD rate limits
Page of 52 71 Page of 52 71
if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 5
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle
after_count =gt 5
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 5
period =gt 1200
key =gt servicename_alarm_warning
add_tag =gt servicename_throttled
Page of 53 71 Page of 53 71
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
output
elasticsearch
hosts =gt localhost
stdout codec =gt rubydebug
if servicename_alarm_err in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
Page of 54 71 Page of 54 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 55 71 Page of 55 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
Page of 56 71 Page of 56 71
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert err message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_error in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
Page of 57 71 Page of 57 71
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 58 71 Page of 58 71
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 59 71 Page of 59 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert error message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
Page of 60 71 Page of 60 71
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
Page of 61 71 Page of 61 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 62 71 Page of 62 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt x
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert fatal message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
Page of 63 71 Page of 63 71
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
Page of 64 71 Page of 64 71
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 65 71 Page of 65 71
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt heavy_exclamation_mark
Page of 66 71 Page of 66 71
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert WARNING message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 67 71 Page of 67 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
Page of 68 71 Page of 68 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 69 71 Page of 69 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
Page of 70 71 Page of 70 71
Page of 71 71 Page of 71 71
13
Transfer13 the13 files13 to13 the13 virtual13 server13 (using13 the13 scp13 method13 detailed13 above)13
Install13 Elas0csearch13 As13 a13 root13 user13 create13 a13 non-shy‐root13 user13 with13 the13 adduser13 command13
Create13 a13 directory13 to13 install13 Elascsearch13 into13 and13 untar13 the13 contents13 of13 the13 Elascsearch13 install13 file13 into13 that13 directory13
[elkelkforcase]$ cd home
[elkelkforcase]$ mkdir elasticsearch
[elkelkforcase]$ cp homeibmcloudelasticsearch-233targz homeelasticsearch
[elkelkforcase]$ cd elasticsearch
[elkelkforcase]$ gunzip elasticsearch-233targz
[elkelkforcase]$ untar elasticsearch-233tar
Elascsearch13 is13 now13 installed13
Start13 Elas0csearch13 To13 start13 Elascsearch13 go13 to13 the13 bin13 directory13 as13 the13 non-shy‐root13 user13 when13 you13 installed13 it13 and13 run13 the13 elascsearch13 executable13
[elkelkforcase]$ cd homeelasticsearchelasticsearch-233bin
[elkelkforcase]$ elasticsearch amp
[2] 22211
[elkelkforcase]$ [2016-07-05 133857730][INFO ][node] [Arsenal] version[233] pid[22211] build[218bdf12016-05-17T154004Z]
[2016-07-05 133857740][INFO ][node] [Arsenal] initializing
[2016-07-05 133858984][INFO ][plugins] [Arsenal] modules [reindex lang-expression lang-groovy] plugins [] sites []
[2016-07-05 133859146][INFO ][env] [Arsenal] using [1] data paths mounts [[ (rootfs)]] net usable_space [155gb] net total_space [199gb] spins [unknown] types [rootfs]
[2016-07-05 133859146][INFO ][env] [Arsenal] heap size [10156mb] compressed ordinary object pointers [true]
Page of 14 71 Page of 14 71
[2016-07-05 133859147][WARN ][env] [Arsenal] max file descriptors [4096] for elasticsearch process likely too low consider increasing to at least [65536]
[2016-07-05 133903526][INFO ][node] [Arsenal] initialized
[2016-07-05 133903526][INFO ][node] [Arsenal] starting
To13 test13 that13 Elascsearch13 is13 running13 correctly13 open13 the13 browser13 in13 the13 VNC13 client13 and13 run13 localhost920013
13
Install13 Kibana13 ELK13 recommends13 installing13 and13 running13 Kibana13 as13 root13 13
1 As13 root13 user13 create13 an13 installaon13 directory13 and13 untar13 the13 installaon13 file13
[rootelkforcase]$ cd home
[rootelkforcase]$ mkdir kibana
[rootelkforcase]$ cp homeibmcloudkibana-451-linux-x64targz homekibana
[rootelkforcase]$ cd kibana
[rootelkforcase]$ gunzip kibana-451-linux-x64targz
[rootelkforcase]$ untar kibana-451-linux-x64tar
2 Modify13 the13 manifestyml13 configuraon13 file13 Go13 to13 the13 kibana-shy‐451-shy‐linux-shy‐x6413 directory13
[rootelkforcase]$ cd homekibanakibana-451-linux-x64
3 Edit13 the13 manifestyml13 file13 so13 the13 contents13 are13
applicaons13
-shy‐ name13 kibana-shy‐in-shy‐bluemix13
memory13 128M13
host13 kibana-shy‐in-shy‐bluemix13
buildpack13 hcpsgithubcomcloudfoundry-shy‐communitynginx-shy‐buildpackgit13
Page of 15 71 Page of 15 71
4 Edit13 the13 configkibanayml13 file13 so13 the13 contents13 are13
Kibana is served by a back end server This controls which port to use
port 5601
The host to bind the server to
host 0000
If you are running kibana behind a proxy and want to mount it at a path
specify that path here The basePath cant end in a slash
serverbasePath
The maximum payload size in bytes on incoming server requests
servermaxPayloadBytes 1048576
The Elasticsearch instance to use for all your queries
elasticsearchurl httplocalhost9200
preserve_elasticsearch_host true will send the hostname specified in `elasticsearch` If you set it to false
then the host you use to connect to this Kibana instance will be sent
elasticsearchpreserveHost true
Kibana uses an index in Elasticsearch to store saved searches visualizations
and dashboards It will create a new index if it doesnt already exist
kibanaindex kibana
The default application to load
kibanadefaultAppId discover
If your Elasticsearch is protected with basic auth these are the user credentials
used by the Kibana server to perform maintenance on the kibana_index at startup Your
Kibana users will still need to authenticate with Elasticsearch (which is proxied through
the Kibana server)
Page of 16 71 Page of 16 71
elasticsearchusername user
elasticsearchpassword pass
SSL for outgoing requests from the Kibana Server to the browser (PEM formatted)
serversslcert pathtoyourservercrt
serversslkey pathtoyourserverkey
Optional setting to validate that your Elasticsearch backend uses the same key files (PEM formatted)
elasticsearchsslcert pathtoyourclientcrt
elasticsearchsslkey pathtoyourclientkey
If you need to provide a CA certificate for your Elasticsearch instance put
the path of the pem file here
elasticsearchsslca pathtoyourCApem
Set to false to have a complete disregard for the validity of the SSL
certificate
elasticsearchsslverify true
Time in milliseconds to wait for elasticsearch to respond to pings defaults to
request_timeout setting
elasticsearchpingTimeout 1500
Time in milliseconds to wait for responses from the back end or elasticsearch
This must be gt 0
elasticsearchrequestTimeout 30000
Time in milliseconds for Elasticsearch to wait for responses from shards
Set to 0 to disable
elasticsearchshardTimeout 0
Time in milliseconds to wait for Elasticsearch at Kibana startup before retrying
elasticsearchstartupTimeout 5000
elasticsearchstartupTimeout 30000
Page of 17 71 Page of 17 71
Set the path to where you would like the process id file to be created
pidfile varrunkibanapid
If you would like to send the log output to a file you can set the path below
loggingdest stdout
Set this to true to suppress all logging output
loggingsilent false
Set this to true to suppress all logging output except for error messages
loggingquiet false
Set this to true to log all events including system usage information and all requests
loggingverbose false
Kibana13 is13 now13 installed13
Start13 Kibana13 To13 start13 Kibana13 go13 to13 the13 bin13 directory13 and13 run13 the13 kibana13 executable13
[rootelkforcase]$
[rootelkforcase]$ cd homekibanakibana-451-linux-x64bin
[rootelkforcase]$ kibana amp
[1] 22283
[rootelkforcase] log [095900542] [info][status][pluginkibana] Status changed from uninitialized to green - Ready
log [095900600] [info][status][pluginelasticsearch] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [095900689] [info][status][pluginkbn_vislib_vis_types] Status changed from uninitialized to green - Ready
log [095900717] [info][status][pluginmarkdown_vis] Status changed from uninitialized to green - Ready
log [095900721] [info][status][pluginmetric_vis] Status changed from uninitialized to green - Ready
log [095900814] [info][status][pluginspyModes] Status changed from uninitialized to green - Ready
log [095900934] [info][status][pluginstatusPage] Status changed from uninitialized to green - Ready
Page of 18 71 Page of 18 71
log [095900941] [info][status][plugintable_vis] Status changed from uninitialized to green - Ready
log [095901067] [info][listening] Server running at http00005601
log [095906083] [info][status][pluginelasticsearch] Status changed from yellow to yellow - No existing Kibana index found
log [095908942] [info][status][pluginelasticsearch] Status changed from yellow to green - Kibana index ready
Kibana13 is13 now13 started13
Broadcast13 Bluemix13 app13 log13 records13 Bluemix13 apps13 (which13 are13 shown13 in13 the13 US13 South13 Bluemix13 region13 of13 the13 System13 Architecture13 diagram)13 will13 produce13 logs13 in13 a13 syslog13 rfc542413 format13 using13 a13 Bluemix13 tool13 called13 Loggregator13
For13 Logstash13 to13 receive13 these13 logs13 each13 Bluemix13 app13 needs13 to13 broadcast13 these13 logs13 To13 do13 that13 run13 the13 following13 commands13 on13 each13 Bluemix13 app13 using13 the13 public13 IP13 address13 of13 the13 ELK13 virtual13 server13 the13 ELK13 tag13 to13 trace13 the13 process13 and13 insert13 the13 name13 of13 each13 app13 in13 turn13
cf cups elk -l syslogltpublicIPaddress5000 cf bind-service ltmyappgt elk
cf restage ltmyappgt
Install13 Logstash13 To13 install13 Logstash13 as13 root13 user13 create13 a13 directory13 for13 the13 installaon13 and13 untar13 the13 installaon13 file13 there13
13 13 [rootelkforcase]$ cd home
[rootelkforcase]$ mkdir logstash
[rootelkforcase]$ cp homeibmcloudlogstash-233targz homelogstash
[rootelkforcase]$ cd logstash
[rootelkforcase]$ gunzip logstash-233targz
[rootelkforcase]$ untar logstash-233tar
Write13 Logstash13 configura0on13 Logstash13 requires13 a13 configuraon13 file13 to13 be13 wricen13 to13 receive13 transform13 and13 output13 the13 log13 records13 to13 Elascsearch13 This13 secon13 explains13 the13 basic13 requirements13 of13 processing13 Bluemix13 app13 log13 records13 into13 Elascsearch13 (more13 complex13 configuraons13 are13 detailed13 later13 in13 this13 document13 for13 the13 integraons13 with13 third-shy‐party13 tools)13
Create13 a13 conf13 file13 as13 root13 user13
[rootelkforcase]$ cd homelogstashlogstash-233
[rootelkforcase]$ mkdir config
Page of 19 71 Page of 19 71
[rootelkforcase]$ cd config
[rootelkforcase]$ touch loggregatorconf
Edit13 the13 loggregator13 file13 with13 the13 following13 contents13
13 Input13 secon13 defines13 which13 log13 records13 to13 receive13
13 port13 and13 type13 reflect13 method13 selected13 for13 Cloud13 Foundry13 to13 broadcast13 the13 records13
input
tcp
port =gt 5000
type =gt syslog
13 Filter13 secon13 ndash13 transforms13 the13 data13
filter
Add a field to identify the records as Cloud Foundry based
mutate
add_field =gt format =gt cf
13 Output13 secon13 ndash13 sends13 the13 data13 to13 its13 desnaon(s)13
output
13 Use13 the13 elascsearch13 plugin13 to13 send13 the13 data13 to13 the13 elascsearch13 instance13
13 elasticsearch
hosts =gt localhost
stdout codec =gt rubydebug
13 End13 of13 Config13
Save13 the13 file13 Verify13 the13 format13 of13 the13 file13 is13 acceptable13 by13 running13 the13 following13 commands13
[rootelkforcase]$ cd homelogstashlogstash-233
[rootelkforcase]$ binlogstash agent --configtest -f configloggregatorconf
Page of 20 71 Page of 20 71
Configuraon13 is13 OK13
Start13 Logstash13 Aaer13 you13 have13 wricen13 and13 verified13 the13 Logstash13 configuraon13 you13 can13 start13 Logstash13 to13 process13 records13 from13 the13 Bluemix13 apps13 (via13 Cloud13 Foundry)13 Run13 the13 following13 as13 root13 user13
[rootelkforcase]$ cd homelogstashlogstash-233
[rootelkforcase]$ binlogstash agent -f configloggregatorconf
Logstash13 is13 now13 started13
Monitor13 Logstash13 processing13 Once13 Logstash13 is13 running13 and13 log13 records13 are13 being13 produced13 by13 the13 Bluemix13 apps13 you13 can13 monitor13 progress13 in13 the13 same13 window13 that13 was13 used13 to13 start13 Logstash13 Output13 is13 similar13 to13 the13 following13
13
The13 log13 records13 being13 broadcast13 from13 the13 Bluemix13 apps13 are13 now13 sent13 to13 Elascsearch13 from13 the13 Logstash13 output13 configuraon13 Records13 are13 sent13 in13 real13 me13 If13 Logstash13 is13 not13 running13 while13 log13 records13 are13 being13 produced13 then13 these13 records13 will13 not13 be13 processed13 by13 Logstash13 and13 will13 not13 be13 stored13 in13 Elascsearch13
Page of 21 71 Page of 21 71
Use13 ELK13 The13 log13 records13 from13 the13 Bluemix13 apps13 are13 now13 being13 stored13 and13 indexed13 by13 the13 Elascsearch13 component13 This13 data13 can13 be13 accessed13 via13 the13 Kibana13 applicaon13
Kibana13 Discover13 page13 Using13 a13 web13 browser13 either13 directly13 from13 a13 client13 or13 using13 the13 VNC13 client13 access13 Kibana13 using13 the13 publicIPaddress13 of13 the13 ELK13 virtual13 server13
hcpltpublicIPaddressgt560113
No13 userpassword13 credenals13 are13 required13
Kibana13 defaults13 to13 a13 search13 of13 all13 records13 ()13 in13 the13 last13 1513 minutes13 and13 returns13 the13 results13
13
More13 complex13 searches13 are13 possible13 by13 changing13 the13 me13 filter13 and13 supplying13 search13 queries13
Change13 the13 0me13 filter13 To13 change13 the13 me13 ranges13 click13 the13 clock13 symbol13 in13 the13 top-shy‐right13 of13 the13 Kibana13 Discover13 page13
13
You13 will13 see13 the13 following13 opons13
13
Page of 22 71 Page of 22 71
A13 variety13 of13 quick13 me13 ranges13 can13 be13 selected13 or13 you13 can13 switch13 to13 relave13 or13 absolute13 me13 ranges13
Quick13 me13 ranges13 allow13 switching13 between13 last13 X13 minutes13 to13 last13 Y13 hours13 or13 Z13 days13 etc13
Relave13 me13 ranges13 allow13 more13 specific13 versions13 of13 the13 Quick13 opons13 from13 specific13 minuteshoursdays13 to13 now13
13
Absolute13 me13 ranges13 specify13 the13 start13 and13 end13 mes13 for13 the13 searches13
13
Custom13 search13 queries13 You13 can13 use13 the13 text13 window13 of13 the13 Discover13 page13 to13 search13 for13 matches13 within13 the13 log13 records13 You13 can13 use13 Apache13 Lucene13 or13 Elascsearch13 query13 DSL13 syntax13
13
Page of 23 71 Page of 23 71
Saved13 searches13 You13 can13 save13 any13 custom13 search13 syntax13 and13 choice13 of13 me13 range13 for13 future13 ease13 of13 use13 Any13 relave13 or13 absolute13 me13 frame13 is13 saved13 as13 part13 of13 the13 search13 13
To13 save13 a13 search13
1 Set13 up13 and13 run13 the13 required13 syntax13 and13 me13 frame13
13
2 Click13 the13 Save13 Search13 icon13
13
3 Type13 in13 a13 name13 for13 the13 saved13 search13
Page of 24 71 Page of 24 71
13
7 To13 delete13 or13 update13 saved13 searches13 click13 13 manage13 saved13 searches13
13
Use13 the13 edit13 icons13 to13 make13 any13 changes13
Page of 26 71 Page of 26 71
13
Build13 simple13 Kibana13 histograms13 Kibana13 allows13 you13 to13 build13 charts13 and13 dashboards13 based13 on13 the13 search13 capability13 offered13 by13 the13 Discover13 page13 Saved13 searches13 can13 be13 used13 as13 the13 basis13 for13 charts13 and13 dashboards13 built13 using13 those13 charts13 To13 build13 a13 simple13 histogram13
1 Select13 the13 Visualize13 page13
13
2 Select13 Ver0cal13 Bar13 Chart13
13
3 To13 use13 a13 saved13 search13 click13 From13 a13 saved13 search13
Page of 27 71 Page of 27 71
13
4 Select13 a13 previously13 saved13 search13
13
5 Select13 to13 edit13 the13 x-shy‐axis
6 Edit13 the13 fields13 with13 informaon13 about13 the13 aggregaon13 field13 interval13 and13 custom13 label13
Page of 28 71 Page of 28 71
13
Page of 29 71 Page of 29 71
7 Click13 the13 green13 arrow13 The13 histogram13 is13 created13 and13 run13 based13 on13 the13 saved13 search
13
8 To13 save13 the13 histogram13 click13 the13 Save13 Visualiza0on13 icon13 and13 add13 a13 tle13
13
13
9 Click13 Save13
10 To13 view13 the13 visualisaon13 click13 the13 Dashboard13 page13
Page of 30 71 Page of 30 71
13
11 Click13 the13 Add13 Visualiza0on13 icon13
13
12 Select13 the13 visiualizaon13 that13 you13 just13 created13
13
Page of 31 71 Page of 31 71
13
13 To13 save13 the13 dashboard13 click13 the13 Save13 Dashboard13 icon13
Page of 32 71 Page of 32 71
13
14 Add13 a13 tle13 and13 select13 whether13 to13 store13 the13 selected13 me13 frame13 with13 the13 dashboard13
13
15 Click13 Save13
16 You13 can13 edit13 the13 shape13 and13 number13 of13 charts13 and13 save13 the13 dashboard13
17 To13 re-shy‐run13 the13 dashboard13 select13 the13 Dashboard13 page13 and13 click13 the13 Load13 Saved13 Dashboard13 icon13
Page of 33 71 Page of 33 71
13
18 Select13 the13 saved13 dashboard13 from13 the13 list13
13
19 The13 dashboard13 runs13
13
Page of 34 71 Page of 34 71
Page of 35 71 Page of 35 71
Aler0ng13 with13 ELK13 ELK13 examines13 the13 log13 records13 in13 process13 to13 send13 alerts13 to13 third-shy‐party13 alerng13 event13 management13 and13 social13 media13 applicaons13 As13 shown13 previously13 in13 the13 ELK13 architecture13 the13 ELK13 strategy13 for13 alerng13 is13 to13 send13 alerts13 from13 the13 Logstash13 component13 directly13 Therefore13 log13 records13 and13 alerts13 are13 sent13 to13 both13 Elascsearch13 and13 third-shy‐party13 tools13 at13 the13 same13 instant13 using13 the13 output13 secon13 of13 the13 Logstash13 configuraon13 Any13 alerts13 are13 generated13 during13 the13 filter13 secon13 processing13 of13 the13 Logstash13 configuraon13
Create13 alerts13 in13 Logstash13 The13 Logstash13 filter13 secon13 is13 used13 to13 trigger13 alerts13 based13 on13 the13 log13 records13 being13 processed13 For13 example13 log13 records13 can13 be13 checked13 for13 errors13 warnings13 crashed13 or13 stopped13 messages13 and13 generate13 an13 alert13
Here13 is13 an13 example13 Logstash13 filter13 code13
13 if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
The13 specific13 tags13 are13 set13 against13 the13 log13 records13 to13 trigger13 the13 alert13 later13 in13 the13 configuraon13 in13 the13 output13 secon13
Control13 the13 number13 of13 generated13 alerts13 using13 the13 Logstash13 throcle13 plugin13 so13 that13 either13 a13 lot13 of13 instances13 in13 a13 fixed13 me13 period13 are13 required13 before13 an13 alert13 is13 sent13 or13 a13 specific13 number13 of13 instances13 triggers13 the13 alert13
Page of 36 71 Page of 36 71
Edit13 the13 plugin13 parameters13 to13 meet13 system13 requirements13
13 13 13 13 13 if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 10
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle13
after_count =gt 20
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 30
period =gt 1200
key =gt servicename_alarm_warning
Page of 37 71 Page of 37 71
add_tag =gt servicename_throttled
13 13 13 13 13 13 13 13 13
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 40
period =gt 1500
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
Enrich13 alerts13 with13 message13 detail13 Some13 of13 the13 third-shy‐party13 integraons13 benefit13 from13 having13 components13 of13 the13 raw13 log13 record13 sent13 in13 addion13 to13 the13 raw13 record13 or13 fields13 in13 the13 record13 edited13 transformed13 or13 translated13
Grok13 Logstash13 allows13 the13 log13 record13 to13 be13 broken13 up13 into13 logical13 secons13 and13 each13 secon13 stored13 in13 a13 new13 parameter13 using13 the13 grok13 parameter13 in13 the13 filter13 secon13 Here13 is13 an13 example13 grok13
13 grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
To13 build13 the13 grok13 statement13 Logstash13 offers13 an13 online13 tool13 at13 hcpsgrokdebugherokuappcom13
Translate13 If13 specific13 fields13 in13 the13 log13 record13 are13 coded13 (for13 example13 the13 Bluemix13 app13 name13 is13 oaen13 not13 sent13 in13 the13 syslog13 messages13 instead13 the13 GUID13 is13 sent13 which13 is13 not13 useful)13 then13 the13 filter13 secon13 translate13 plugin13 can13 be13 used13 to13 add13 new13 fields13 by13 holding13 the13 decoded13 versions13 of13 those13 values13
13 translate
field =gt data1
destination =gt data1
override =gt true
Page of 38 71 Page of 38 71
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
dc50f6e4-c052-436a-bf27-95ba392c4228 monverifyservicemanagement
0c677e82-5bfc-4e38-9155-e416249e3ee6 ms-catalog-postpyretic-pomiculture
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
However13 the13 translate13 plugin13 is13 not13 included13 with13 the13 Logstash13 installaon13 In13 the13 next13 secon13 I13 will13 13 describe13 how13 to13 add13 new13 plugins13 to13 the13 Logstash13 instance13
Add13 custom13 plugins13 to13 Logstash13 As13 an13 open13 source13 tool13 Logstash13 encourages13 users13 to13 develop13 content13 for13 all13 users13 to13 share13 Some13 of13 the13 plugins13 that13 were13 created13 by13 our13 community13 include13 a13 translaon13 plugin13 an13 output13 Slack13 plugin13 and13 an13 output13 syslog13 plugin13 Letrsquos13 look13 at13 how13 to13 add13 these13 plugins13 to13 your13 environment13
Filter13 translate13 plugin13 To13 add13 the13 translate13 plugin13
1 Download13 the13 plugin13
2 Copy13 the13 plugin13 file13 logstash-shy‐filter-shy‐translate-shy‐214gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-filter-translate 214
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Output13 Slack13 plugin13 To13 add13 the13 Slack13 plugin13
Page of 39 71 Page of 39 71
1 Download13 the13 plugin13 13
2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐slack-shy‐014gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-output-slack 014
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Output13 syslog13 plugin13 To13 add13 the13 syslog13 plugin13
1 Download13 the13 plugin13 13
2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐syslog-shy‐214gem13 to13 the13 ELK13 Virtual13 Server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-output-syslog 214
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Integrate13 Logstash13 with13 Slack13 To13 send13 generated13 alerts13 to13 Slack13 complete13 these13 steps13
1 Create13 a13 Slack13 integraon13 13
2 Use13 the13 Slack13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 Slack13 integraon13
113 Login13 to13 Slack13 and13 go13 to13 the13 Apps13 page13 13
Page of 40 71 Page of 40 71
13
313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13
13
A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13
13
Page of 42 71 Page of 42 71
The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13
13
The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13
13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
slack
url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5
channel =gt cloud-operations-mon
username =gt nick_cawood
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13
Page of 43 71 Page of 43 71
Page of 44 71 Page of 44 71
In13 Slack13 the13 alerts13 look13 like13 this13
13
Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13
1 Create13 a13 PagerDuty13 integraon13 13
2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13
1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13
13
2 Click13 Add13 New13 Service13
13
313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13
Page of 45 71 Page of 45 71
13
413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13
513 Click13 Add13 Service13
13
The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13
Page of 46 71 Page of 46 71
13
The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt be7eb1ea9c224018923e818d3e891d0b
incident_key =gt logstashhosttype
Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13
Page of 47 71 Page of 47 71
13
Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13
1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13
2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13
3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13
4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13
13 13 http
url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo
http_method =gt post
format =gt json
For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13
13 13 syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
Page of 48 71 Page of 48 71
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13
13
The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13
if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2
This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13
Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13
Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13
Logstash13 severity Shown13 in13 Omnibus13 as
alert Indeterminate
crical Warning
debug 7
emergency Clear
error Minor
informaonal 6
noce Crical
warning Major
Page of 49 71 Page of 49 71
13
Page of 50 71 Page of 50 71
Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13
input
tcp
port =gt 5000
type =gt syslog
filter
mutate
add_field =gt format =gt cf
grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
translate
field =gt data1
destination =gt data1
override =gt true
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
=== Check Alarm Criteria ===
Standard Key Words
if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
=== Alarm Exclusions ===
if [message] =~ (i)sometexthere
mutate remove_tag =gt servicename_alarm
Throttle alarms so we do not hit PD rate limits
Page of 52 71 Page of 52 71
if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 5
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle
after_count =gt 5
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 5
period =gt 1200
key =gt servicename_alarm_warning
add_tag =gt servicename_throttled
Page of 53 71 Page of 53 71
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
output
elasticsearch
hosts =gt localhost
stdout codec =gt rubydebug
if servicename_alarm_err in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
Page of 54 71 Page of 54 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 55 71 Page of 55 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
Page of 56 71 Page of 56 71
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert err message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_error in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
Page of 57 71 Page of 57 71
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 58 71 Page of 58 71
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 59 71 Page of 59 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert error message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
Page of 60 71 Page of 60 71
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
Page of 61 71 Page of 61 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 62 71 Page of 62 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt x
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert fatal message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
Page of 63 71 Page of 63 71
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
Page of 64 71 Page of 64 71
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 65 71 Page of 65 71
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt heavy_exclamation_mark
Page of 66 71 Page of 66 71
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert WARNING message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 67 71 Page of 67 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
Page of 68 71 Page of 68 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 69 71 Page of 69 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
Page of 70 71 Page of 70 71
Page of 71 71 Page of 71 71
[2016-07-05 133859147][WARN ][env] [Arsenal] max file descriptors [4096] for elasticsearch process likely too low consider increasing to at least [65536]
[2016-07-05 133903526][INFO ][node] [Arsenal] initialized
[2016-07-05 133903526][INFO ][node] [Arsenal] starting
To13 test13 that13 Elascsearch13 is13 running13 correctly13 open13 the13 browser13 in13 the13 VNC13 client13 and13 run13 localhost920013
13
Install13 Kibana13 ELK13 recommends13 installing13 and13 running13 Kibana13 as13 root13 13
1 As13 root13 user13 create13 an13 installaon13 directory13 and13 untar13 the13 installaon13 file13
[rootelkforcase]$ cd home
[rootelkforcase]$ mkdir kibana
[rootelkforcase]$ cp homeibmcloudkibana-451-linux-x64targz homekibana
[rootelkforcase]$ cd kibana
[rootelkforcase]$ gunzip kibana-451-linux-x64targz
[rootelkforcase]$ untar kibana-451-linux-x64tar
2 Modify13 the13 manifestyml13 configuraon13 file13 Go13 to13 the13 kibana-shy‐451-shy‐linux-shy‐x6413 directory13
[rootelkforcase]$ cd homekibanakibana-451-linux-x64
3 Edit13 the13 manifestyml13 file13 so13 the13 contents13 are13
applicaons13
-shy‐ name13 kibana-shy‐in-shy‐bluemix13
memory13 128M13
host13 kibana-shy‐in-shy‐bluemix13
buildpack13 hcpsgithubcomcloudfoundry-shy‐communitynginx-shy‐buildpackgit13
Page of 15 71 Page of 15 71
4 Edit13 the13 configkibanayml13 file13 so13 the13 contents13 are13
Kibana is served by a back end server This controls which port to use
port 5601
The host to bind the server to
host 0000
If you are running kibana behind a proxy and want to mount it at a path
specify that path here The basePath cant end in a slash
serverbasePath
The maximum payload size in bytes on incoming server requests
servermaxPayloadBytes 1048576
The Elasticsearch instance to use for all your queries
elasticsearchurl httplocalhost9200
preserve_elasticsearch_host true will send the hostname specified in `elasticsearch` If you set it to false
then the host you use to connect to this Kibana instance will be sent
elasticsearchpreserveHost true
Kibana uses an index in Elasticsearch to store saved searches visualizations
and dashboards It will create a new index if it doesnt already exist
kibanaindex kibana
The default application to load
kibanadefaultAppId discover
If your Elasticsearch is protected with basic auth these are the user credentials
used by the Kibana server to perform maintenance on the kibana_index at startup Your
Kibana users will still need to authenticate with Elasticsearch (which is proxied through
the Kibana server)
Page of 16 71 Page of 16 71
elasticsearchusername user
elasticsearchpassword pass
SSL for outgoing requests from the Kibana Server to the browser (PEM formatted)
serversslcert pathtoyourservercrt
serversslkey pathtoyourserverkey
Optional setting to validate that your Elasticsearch backend uses the same key files (PEM formatted)
elasticsearchsslcert pathtoyourclientcrt
elasticsearchsslkey pathtoyourclientkey
If you need to provide a CA certificate for your Elasticsearch instance put
the path of the pem file here
elasticsearchsslca pathtoyourCApem
Set to false to have a complete disregard for the validity of the SSL
certificate
elasticsearchsslverify true
Time in milliseconds to wait for elasticsearch to respond to pings defaults to
request_timeout setting
elasticsearchpingTimeout 1500
Time in milliseconds to wait for responses from the back end or elasticsearch
This must be gt 0
elasticsearchrequestTimeout 30000
Time in milliseconds for Elasticsearch to wait for responses from shards
Set to 0 to disable
elasticsearchshardTimeout 0
Time in milliseconds to wait for Elasticsearch at Kibana startup before retrying
elasticsearchstartupTimeout 5000
elasticsearchstartupTimeout 30000
Page of 17 71 Page of 17 71
Set the path to where you would like the process id file to be created
pidfile varrunkibanapid
If you would like to send the log output to a file you can set the path below
loggingdest stdout
Set this to true to suppress all logging output
loggingsilent false
Set this to true to suppress all logging output except for error messages
loggingquiet false
Set this to true to log all events including system usage information and all requests
loggingverbose false
Kibana13 is13 now13 installed13
Start13 Kibana13 To13 start13 Kibana13 go13 to13 the13 bin13 directory13 and13 run13 the13 kibana13 executable13
[rootelkforcase]$
[rootelkforcase]$ cd homekibanakibana-451-linux-x64bin
[rootelkforcase]$ kibana amp
[1] 22283
[rootelkforcase] log [095900542] [info][status][pluginkibana] Status changed from uninitialized to green - Ready
log [095900600] [info][status][pluginelasticsearch] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [095900689] [info][status][pluginkbn_vislib_vis_types] Status changed from uninitialized to green - Ready
log [095900717] [info][status][pluginmarkdown_vis] Status changed from uninitialized to green - Ready
log [095900721] [info][status][pluginmetric_vis] Status changed from uninitialized to green - Ready
log [095900814] [info][status][pluginspyModes] Status changed from uninitialized to green - Ready
log [095900934] [info][status][pluginstatusPage] Status changed from uninitialized to green - Ready
Page of 18 71 Page of 18 71
log [095900941] [info][status][plugintable_vis] Status changed from uninitialized to green - Ready
log [095901067] [info][listening] Server running at http00005601
log [095906083] [info][status][pluginelasticsearch] Status changed from yellow to yellow - No existing Kibana index found
log [095908942] [info][status][pluginelasticsearch] Status changed from yellow to green - Kibana index ready
Kibana13 is13 now13 started13
Broadcast13 Bluemix13 app13 log13 records13 Bluemix13 apps13 (which13 are13 shown13 in13 the13 US13 South13 Bluemix13 region13 of13 the13 System13 Architecture13 diagram)13 will13 produce13 logs13 in13 a13 syslog13 rfc542413 format13 using13 a13 Bluemix13 tool13 called13 Loggregator13
For13 Logstash13 to13 receive13 these13 logs13 each13 Bluemix13 app13 needs13 to13 broadcast13 these13 logs13 To13 do13 that13 run13 the13 following13 commands13 on13 each13 Bluemix13 app13 using13 the13 public13 IP13 address13 of13 the13 ELK13 virtual13 server13 the13 ELK13 tag13 to13 trace13 the13 process13 and13 insert13 the13 name13 of13 each13 app13 in13 turn13
cf cups elk -l syslogltpublicIPaddress5000 cf bind-service ltmyappgt elk
cf restage ltmyappgt
Install13 Logstash13 To13 install13 Logstash13 as13 root13 user13 create13 a13 directory13 for13 the13 installaon13 and13 untar13 the13 installaon13 file13 there13
13 13 [rootelkforcase]$ cd home
[rootelkforcase]$ mkdir logstash
[rootelkforcase]$ cp homeibmcloudlogstash-233targz homelogstash
[rootelkforcase]$ cd logstash
[rootelkforcase]$ gunzip logstash-233targz
[rootelkforcase]$ untar logstash-233tar
Write13 Logstash13 configura0on13 Logstash13 requires13 a13 configuraon13 file13 to13 be13 wricen13 to13 receive13 transform13 and13 output13 the13 log13 records13 to13 Elascsearch13 This13 secon13 explains13 the13 basic13 requirements13 of13 processing13 Bluemix13 app13 log13 records13 into13 Elascsearch13 (more13 complex13 configuraons13 are13 detailed13 later13 in13 this13 document13 for13 the13 integraons13 with13 third-shy‐party13 tools)13
Create13 a13 conf13 file13 as13 root13 user13
[rootelkforcase]$ cd homelogstashlogstash-233
[rootelkforcase]$ mkdir config
Page of 19 71 Page of 19 71
[rootelkforcase]$ cd config
[rootelkforcase]$ touch loggregatorconf
Edit13 the13 loggregator13 file13 with13 the13 following13 contents13
13 Input13 secon13 defines13 which13 log13 records13 to13 receive13
13 port13 and13 type13 reflect13 method13 selected13 for13 Cloud13 Foundry13 to13 broadcast13 the13 records13
input
tcp
port =gt 5000
type =gt syslog
13 Filter13 secon13 ndash13 transforms13 the13 data13
filter
Add a field to identify the records as Cloud Foundry based
mutate
add_field =gt format =gt cf
13 Output13 secon13 ndash13 sends13 the13 data13 to13 its13 desnaon(s)13
output
13 Use13 the13 elascsearch13 plugin13 to13 send13 the13 data13 to13 the13 elascsearch13 instance13
13 elasticsearch
hosts =gt localhost
stdout codec =gt rubydebug
13 End13 of13 Config13
Save13 the13 file13 Verify13 the13 format13 of13 the13 file13 is13 acceptable13 by13 running13 the13 following13 commands13
[rootelkforcase]$ cd homelogstashlogstash-233
[rootelkforcase]$ binlogstash agent --configtest -f configloggregatorconf
Page of 20 71 Page of 20 71
Configuraon13 is13 OK13
Start13 Logstash13 Aaer13 you13 have13 wricen13 and13 verified13 the13 Logstash13 configuraon13 you13 can13 start13 Logstash13 to13 process13 records13 from13 the13 Bluemix13 apps13 (via13 Cloud13 Foundry)13 Run13 the13 following13 as13 root13 user13
[rootelkforcase]$ cd homelogstashlogstash-233
[rootelkforcase]$ binlogstash agent -f configloggregatorconf
Logstash13 is13 now13 started13
Monitor13 Logstash13 processing13 Once13 Logstash13 is13 running13 and13 log13 records13 are13 being13 produced13 by13 the13 Bluemix13 apps13 you13 can13 monitor13 progress13 in13 the13 same13 window13 that13 was13 used13 to13 start13 Logstash13 Output13 is13 similar13 to13 the13 following13
13
The13 log13 records13 being13 broadcast13 from13 the13 Bluemix13 apps13 are13 now13 sent13 to13 Elascsearch13 from13 the13 Logstash13 output13 configuraon13 Records13 are13 sent13 in13 real13 me13 If13 Logstash13 is13 not13 running13 while13 log13 records13 are13 being13 produced13 then13 these13 records13 will13 not13 be13 processed13 by13 Logstash13 and13 will13 not13 be13 stored13 in13 Elascsearch13
Page of 21 71 Page of 21 71
Use13 ELK13 The13 log13 records13 from13 the13 Bluemix13 apps13 are13 now13 being13 stored13 and13 indexed13 by13 the13 Elascsearch13 component13 This13 data13 can13 be13 accessed13 via13 the13 Kibana13 applicaon13
Kibana13 Discover13 page13 Using13 a13 web13 browser13 either13 directly13 from13 a13 client13 or13 using13 the13 VNC13 client13 access13 Kibana13 using13 the13 publicIPaddress13 of13 the13 ELK13 virtual13 server13
hcpltpublicIPaddressgt560113
No13 userpassword13 credenals13 are13 required13
Kibana13 defaults13 to13 a13 search13 of13 all13 records13 ()13 in13 the13 last13 1513 minutes13 and13 returns13 the13 results13
13
More13 complex13 searches13 are13 possible13 by13 changing13 the13 me13 filter13 and13 supplying13 search13 queries13
Change13 the13 0me13 filter13 To13 change13 the13 me13 ranges13 click13 the13 clock13 symbol13 in13 the13 top-shy‐right13 of13 the13 Kibana13 Discover13 page13
13
You13 will13 see13 the13 following13 opons13
13
Page of 22 71 Page of 22 71
A13 variety13 of13 quick13 me13 ranges13 can13 be13 selected13 or13 you13 can13 switch13 to13 relave13 or13 absolute13 me13 ranges13
Quick13 me13 ranges13 allow13 switching13 between13 last13 X13 minutes13 to13 last13 Y13 hours13 or13 Z13 days13 etc13
Relave13 me13 ranges13 allow13 more13 specific13 versions13 of13 the13 Quick13 opons13 from13 specific13 minuteshoursdays13 to13 now13
13
Absolute13 me13 ranges13 specify13 the13 start13 and13 end13 mes13 for13 the13 searches13
13
Custom13 search13 queries13 You13 can13 use13 the13 text13 window13 of13 the13 Discover13 page13 to13 search13 for13 matches13 within13 the13 log13 records13 You13 can13 use13 Apache13 Lucene13 or13 Elascsearch13 query13 DSL13 syntax13
13
Page of 23 71 Page of 23 71
Saved13 searches13 You13 can13 save13 any13 custom13 search13 syntax13 and13 choice13 of13 me13 range13 for13 future13 ease13 of13 use13 Any13 relave13 or13 absolute13 me13 frame13 is13 saved13 as13 part13 of13 the13 search13 13
To13 save13 a13 search13
1 Set13 up13 and13 run13 the13 required13 syntax13 and13 me13 frame13
13
2 Click13 the13 Save13 Search13 icon13
13
3 Type13 in13 a13 name13 for13 the13 saved13 search13
Page of 24 71 Page of 24 71
13
7 To13 delete13 or13 update13 saved13 searches13 click13 13 manage13 saved13 searches13
13
Use13 the13 edit13 icons13 to13 make13 any13 changes13
Page of 26 71 Page of 26 71
13
Build13 simple13 Kibana13 histograms13 Kibana13 allows13 you13 to13 build13 charts13 and13 dashboards13 based13 on13 the13 search13 capability13 offered13 by13 the13 Discover13 page13 Saved13 searches13 can13 be13 used13 as13 the13 basis13 for13 charts13 and13 dashboards13 built13 using13 those13 charts13 To13 build13 a13 simple13 histogram13
1 Select13 the13 Visualize13 page13
13
2 Select13 Ver0cal13 Bar13 Chart13
13
3 To13 use13 a13 saved13 search13 click13 From13 a13 saved13 search13
Page of 27 71 Page of 27 71
13
4 Select13 a13 previously13 saved13 search13
13
5 Select13 to13 edit13 the13 x-shy‐axis
6 Edit13 the13 fields13 with13 informaon13 about13 the13 aggregaon13 field13 interval13 and13 custom13 label13
Page of 28 71 Page of 28 71
13
Page of 29 71 Page of 29 71
7 Click13 the13 green13 arrow13 The13 histogram13 is13 created13 and13 run13 based13 on13 the13 saved13 search
13
8 To13 save13 the13 histogram13 click13 the13 Save13 Visualiza0on13 icon13 and13 add13 a13 tle13
13
13
9 Click13 Save13
10 To13 view13 the13 visualisaon13 click13 the13 Dashboard13 page13
Page of 30 71 Page of 30 71
13
11 Click13 the13 Add13 Visualiza0on13 icon13
13
12 Select13 the13 visiualizaon13 that13 you13 just13 created13
13
Page of 31 71 Page of 31 71
13
13 To13 save13 the13 dashboard13 click13 the13 Save13 Dashboard13 icon13
Page of 32 71 Page of 32 71
13
14 Add13 a13 tle13 and13 select13 whether13 to13 store13 the13 selected13 me13 frame13 with13 the13 dashboard13
13
15 Click13 Save13
16 You13 can13 edit13 the13 shape13 and13 number13 of13 charts13 and13 save13 the13 dashboard13
17 To13 re-shy‐run13 the13 dashboard13 select13 the13 Dashboard13 page13 and13 click13 the13 Load13 Saved13 Dashboard13 icon13
Page of 33 71 Page of 33 71
13
18 Select13 the13 saved13 dashboard13 from13 the13 list13
13
19 The13 dashboard13 runs13
13
Page of 34 71 Page of 34 71
Page of 35 71 Page of 35 71
Aler0ng13 with13 ELK13 ELK13 examines13 the13 log13 records13 in13 process13 to13 send13 alerts13 to13 third-shy‐party13 alerng13 event13 management13 and13 social13 media13 applicaons13 As13 shown13 previously13 in13 the13 ELK13 architecture13 the13 ELK13 strategy13 for13 alerng13 is13 to13 send13 alerts13 from13 the13 Logstash13 component13 directly13 Therefore13 log13 records13 and13 alerts13 are13 sent13 to13 both13 Elascsearch13 and13 third-shy‐party13 tools13 at13 the13 same13 instant13 using13 the13 output13 secon13 of13 the13 Logstash13 configuraon13 Any13 alerts13 are13 generated13 during13 the13 filter13 secon13 processing13 of13 the13 Logstash13 configuraon13
Create13 alerts13 in13 Logstash13 The13 Logstash13 filter13 secon13 is13 used13 to13 trigger13 alerts13 based13 on13 the13 log13 records13 being13 processed13 For13 example13 log13 records13 can13 be13 checked13 for13 errors13 warnings13 crashed13 or13 stopped13 messages13 and13 generate13 an13 alert13
Here13 is13 an13 example13 Logstash13 filter13 code13
13 if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
The13 specific13 tags13 are13 set13 against13 the13 log13 records13 to13 trigger13 the13 alert13 later13 in13 the13 configuraon13 in13 the13 output13 secon13
Control13 the13 number13 of13 generated13 alerts13 using13 the13 Logstash13 throcle13 plugin13 so13 that13 either13 a13 lot13 of13 instances13 in13 a13 fixed13 me13 period13 are13 required13 before13 an13 alert13 is13 sent13 or13 a13 specific13 number13 of13 instances13 triggers13 the13 alert13
Page of 36 71 Page of 36 71
Edit13 the13 plugin13 parameters13 to13 meet13 system13 requirements13
13 13 13 13 13 if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 10
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle13
after_count =gt 20
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 30
period =gt 1200
key =gt servicename_alarm_warning
Page of 37 71 Page of 37 71
add_tag =gt servicename_throttled
13 13 13 13 13 13 13 13 13
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 40
period =gt 1500
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
Enrich13 alerts13 with13 message13 detail13 Some13 of13 the13 third-shy‐party13 integraons13 benefit13 from13 having13 components13 of13 the13 raw13 log13 record13 sent13 in13 addion13 to13 the13 raw13 record13 or13 fields13 in13 the13 record13 edited13 transformed13 or13 translated13
Grok13 Logstash13 allows13 the13 log13 record13 to13 be13 broken13 up13 into13 logical13 secons13 and13 each13 secon13 stored13 in13 a13 new13 parameter13 using13 the13 grok13 parameter13 in13 the13 filter13 secon13 Here13 is13 an13 example13 grok13
13 grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
To13 build13 the13 grok13 statement13 Logstash13 offers13 an13 online13 tool13 at13 hcpsgrokdebugherokuappcom13
Translate13 If13 specific13 fields13 in13 the13 log13 record13 are13 coded13 (for13 example13 the13 Bluemix13 app13 name13 is13 oaen13 not13 sent13 in13 the13 syslog13 messages13 instead13 the13 GUID13 is13 sent13 which13 is13 not13 useful)13 then13 the13 filter13 secon13 translate13 plugin13 can13 be13 used13 to13 add13 new13 fields13 by13 holding13 the13 decoded13 versions13 of13 those13 values13
13 translate
field =gt data1
destination =gt data1
override =gt true
Page of 38 71 Page of 38 71
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
dc50f6e4-c052-436a-bf27-95ba392c4228 monverifyservicemanagement
0c677e82-5bfc-4e38-9155-e416249e3ee6 ms-catalog-postpyretic-pomiculture
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
However13 the13 translate13 plugin13 is13 not13 included13 with13 the13 Logstash13 installaon13 In13 the13 next13 secon13 I13 will13 13 describe13 how13 to13 add13 new13 plugins13 to13 the13 Logstash13 instance13
Add13 custom13 plugins13 to13 Logstash13 As13 an13 open13 source13 tool13 Logstash13 encourages13 users13 to13 develop13 content13 for13 all13 users13 to13 share13 Some13 of13 the13 plugins13 that13 were13 created13 by13 our13 community13 include13 a13 translaon13 plugin13 an13 output13 Slack13 plugin13 and13 an13 output13 syslog13 plugin13 Letrsquos13 look13 at13 how13 to13 add13 these13 plugins13 to13 your13 environment13
Filter13 translate13 plugin13 To13 add13 the13 translate13 plugin13
1 Download13 the13 plugin13
2 Copy13 the13 plugin13 file13 logstash-shy‐filter-shy‐translate-shy‐214gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-filter-translate 214
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Output13 Slack13 plugin13 To13 add13 the13 Slack13 plugin13
Page of 39 71 Page of 39 71
1 Download13 the13 plugin13 13
2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐slack-shy‐014gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-output-slack 014
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Output13 syslog13 plugin13 To13 add13 the13 syslog13 plugin13
1 Download13 the13 plugin13 13
2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐syslog-shy‐214gem13 to13 the13 ELK13 Virtual13 Server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-output-syslog 214
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Integrate13 Logstash13 with13 Slack13 To13 send13 generated13 alerts13 to13 Slack13 complete13 these13 steps13
1 Create13 a13 Slack13 integraon13 13
2 Use13 the13 Slack13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 Slack13 integraon13
113 Login13 to13 Slack13 and13 go13 to13 the13 Apps13 page13 13
Page of 40 71 Page of 40 71
13
313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13
13
A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13
13
Page of 42 71 Page of 42 71
The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13
13
The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13
13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
slack
url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5
channel =gt cloud-operations-mon
username =gt nick_cawood
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13
Page of 43 71 Page of 43 71
Page of 44 71 Page of 44 71
In13 Slack13 the13 alerts13 look13 like13 this13
13
Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13
1 Create13 a13 PagerDuty13 integraon13 13
2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13
1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13
13
2 Click13 Add13 New13 Service13
13
313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13
Page of 45 71 Page of 45 71
13
413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13
513 Click13 Add13 Service13
13
The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13
Page of 46 71 Page of 46 71
13
The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt be7eb1ea9c224018923e818d3e891d0b
incident_key =gt logstashhosttype
Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13
Page of 47 71 Page of 47 71
13
Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13
1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13
2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13
3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13
4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13
13 13 http
url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo
http_method =gt post
format =gt json
For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13
13 13 syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
Page of 48 71 Page of 48 71
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13
13
The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13
if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2
This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13
Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13
Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13
Logstash13 severity Shown13 in13 Omnibus13 as
alert Indeterminate
crical Warning
debug 7
emergency Clear
error Minor
informaonal 6
noce Crical
warning Major
Page of 49 71 Page of 49 71
13
Page of 50 71 Page of 50 71
Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13
input
tcp
port =gt 5000
type =gt syslog
filter
mutate
add_field =gt format =gt cf
grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
translate
field =gt data1
destination =gt data1
override =gt true
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
=== Check Alarm Criteria ===
Standard Key Words
if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
=== Alarm Exclusions ===
if [message] =~ (i)sometexthere
mutate remove_tag =gt servicename_alarm
Throttle alarms so we do not hit PD rate limits
Page of 52 71 Page of 52 71
if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 5
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle
after_count =gt 5
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 5
period =gt 1200
key =gt servicename_alarm_warning
add_tag =gt servicename_throttled
Page of 53 71 Page of 53 71
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
output
elasticsearch
hosts =gt localhost
stdout codec =gt rubydebug
if servicename_alarm_err in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
Page of 54 71 Page of 54 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 55 71 Page of 55 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
Page of 56 71 Page of 56 71
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert err message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_error in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
Page of 57 71 Page of 57 71
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 58 71 Page of 58 71
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 59 71 Page of 59 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert error message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
Page of 60 71 Page of 60 71
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
Page of 61 71 Page of 61 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 62 71 Page of 62 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt x
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert fatal message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
Page of 63 71 Page of 63 71
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
Page of 64 71 Page of 64 71
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 65 71 Page of 65 71
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt heavy_exclamation_mark
Page of 66 71 Page of 66 71
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert WARNING message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 67 71 Page of 67 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
Page of 68 71 Page of 68 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 69 71 Page of 69 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
Page of 70 71 Page of 70 71
Page of 71 71 Page of 71 71
4 Edit13 the13 configkibanayml13 file13 so13 the13 contents13 are13
Kibana is served by a back end server This controls which port to use
port 5601
The host to bind the server to
host 0000
If you are running kibana behind a proxy and want to mount it at a path
specify that path here The basePath cant end in a slash
serverbasePath
The maximum payload size in bytes on incoming server requests
servermaxPayloadBytes 1048576
The Elasticsearch instance to use for all your queries
elasticsearchurl httplocalhost9200
preserve_elasticsearch_host true will send the hostname specified in `elasticsearch` If you set it to false
then the host you use to connect to this Kibana instance will be sent
elasticsearchpreserveHost true
Kibana uses an index in Elasticsearch to store saved searches visualizations
and dashboards It will create a new index if it doesnt already exist
kibanaindex kibana
The default application to load
kibanadefaultAppId discover
If your Elasticsearch is protected with basic auth these are the user credentials
used by the Kibana server to perform maintenance on the kibana_index at startup Your
Kibana users will still need to authenticate with Elasticsearch (which is proxied through
the Kibana server)
Page of 16 71 Page of 16 71
elasticsearchusername user
elasticsearchpassword pass
SSL for outgoing requests from the Kibana Server to the browser (PEM formatted)
serversslcert pathtoyourservercrt
serversslkey pathtoyourserverkey
Optional setting to validate that your Elasticsearch backend uses the same key files (PEM formatted)
elasticsearchsslcert pathtoyourclientcrt
elasticsearchsslkey pathtoyourclientkey
If you need to provide a CA certificate for your Elasticsearch instance put
the path of the pem file here
elasticsearchsslca pathtoyourCApem
Set to false to have a complete disregard for the validity of the SSL
certificate
elasticsearchsslverify true
Time in milliseconds to wait for elasticsearch to respond to pings defaults to
request_timeout setting
elasticsearchpingTimeout 1500
Time in milliseconds to wait for responses from the back end or elasticsearch
This must be gt 0
elasticsearchrequestTimeout 30000
Time in milliseconds for Elasticsearch to wait for responses from shards
Set to 0 to disable
elasticsearchshardTimeout 0
Time in milliseconds to wait for Elasticsearch at Kibana startup before retrying
elasticsearchstartupTimeout 5000
elasticsearchstartupTimeout 30000
Page of 17 71 Page of 17 71
Set the path to where you would like the process id file to be created
pidfile varrunkibanapid
If you would like to send the log output to a file you can set the path below
loggingdest stdout
Set this to true to suppress all logging output
loggingsilent false
Set this to true to suppress all logging output except for error messages
loggingquiet false
Set this to true to log all events including system usage information and all requests
loggingverbose false
Kibana13 is13 now13 installed13
Start13 Kibana13 To13 start13 Kibana13 go13 to13 the13 bin13 directory13 and13 run13 the13 kibana13 executable13
[rootelkforcase]$
[rootelkforcase]$ cd homekibanakibana-451-linux-x64bin
[rootelkforcase]$ kibana amp
[1] 22283
[rootelkforcase] log [095900542] [info][status][pluginkibana] Status changed from uninitialized to green - Ready
log [095900600] [info][status][pluginelasticsearch] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [095900689] [info][status][pluginkbn_vislib_vis_types] Status changed from uninitialized to green - Ready
log [095900717] [info][status][pluginmarkdown_vis] Status changed from uninitialized to green - Ready
log [095900721] [info][status][pluginmetric_vis] Status changed from uninitialized to green - Ready
log [095900814] [info][status][pluginspyModes] Status changed from uninitialized to green - Ready
log [095900934] [info][status][pluginstatusPage] Status changed from uninitialized to green - Ready
Page of 18 71 Page of 18 71
log [095900941] [info][status][plugintable_vis] Status changed from uninitialized to green - Ready
log [095901067] [info][listening] Server running at http00005601
log [095906083] [info][status][pluginelasticsearch] Status changed from yellow to yellow - No existing Kibana index found
log [095908942] [info][status][pluginelasticsearch] Status changed from yellow to green - Kibana index ready
Kibana13 is13 now13 started13
Broadcast13 Bluemix13 app13 log13 records13 Bluemix13 apps13 (which13 are13 shown13 in13 the13 US13 South13 Bluemix13 region13 of13 the13 System13 Architecture13 diagram)13 will13 produce13 logs13 in13 a13 syslog13 rfc542413 format13 using13 a13 Bluemix13 tool13 called13 Loggregator13
For13 Logstash13 to13 receive13 these13 logs13 each13 Bluemix13 app13 needs13 to13 broadcast13 these13 logs13 To13 do13 that13 run13 the13 following13 commands13 on13 each13 Bluemix13 app13 using13 the13 public13 IP13 address13 of13 the13 ELK13 virtual13 server13 the13 ELK13 tag13 to13 trace13 the13 process13 and13 insert13 the13 name13 of13 each13 app13 in13 turn13
cf cups elk -l syslogltpublicIPaddress5000 cf bind-service ltmyappgt elk
cf restage ltmyappgt
Install13 Logstash13 To13 install13 Logstash13 as13 root13 user13 create13 a13 directory13 for13 the13 installaon13 and13 untar13 the13 installaon13 file13 there13
13 13 [rootelkforcase]$ cd home
[rootelkforcase]$ mkdir logstash
[rootelkforcase]$ cp homeibmcloudlogstash-233targz homelogstash
[rootelkforcase]$ cd logstash
[rootelkforcase]$ gunzip logstash-233targz
[rootelkforcase]$ untar logstash-233tar
Write13 Logstash13 configura0on13 Logstash13 requires13 a13 configuraon13 file13 to13 be13 wricen13 to13 receive13 transform13 and13 output13 the13 log13 records13 to13 Elascsearch13 This13 secon13 explains13 the13 basic13 requirements13 of13 processing13 Bluemix13 app13 log13 records13 into13 Elascsearch13 (more13 complex13 configuraons13 are13 detailed13 later13 in13 this13 document13 for13 the13 integraons13 with13 third-shy‐party13 tools)13
Create13 a13 conf13 file13 as13 root13 user13
[rootelkforcase]$ cd homelogstashlogstash-233
[rootelkforcase]$ mkdir config
Page of 19 71 Page of 19 71
[rootelkforcase]$ cd config
[rootelkforcase]$ touch loggregatorconf
Edit13 the13 loggregator13 file13 with13 the13 following13 contents13
13 Input13 secon13 defines13 which13 log13 records13 to13 receive13
13 port13 and13 type13 reflect13 method13 selected13 for13 Cloud13 Foundry13 to13 broadcast13 the13 records13
input
tcp
port =gt 5000
type =gt syslog
13 Filter13 secon13 ndash13 transforms13 the13 data13
filter
Add a field to identify the records as Cloud Foundry based
mutate
add_field =gt format =gt cf
13 Output13 secon13 ndash13 sends13 the13 data13 to13 its13 desnaon(s)13
output
13 Use13 the13 elascsearch13 plugin13 to13 send13 the13 data13 to13 the13 elascsearch13 instance13
13 elasticsearch
hosts =gt localhost
stdout codec =gt rubydebug
13 End13 of13 Config13
Save13 the13 file13 Verify13 the13 format13 of13 the13 file13 is13 acceptable13 by13 running13 the13 following13 commands13
[rootelkforcase]$ cd homelogstashlogstash-233
[rootelkforcase]$ binlogstash agent --configtest -f configloggregatorconf
Page of 20 71 Page of 20 71
Configuraon13 is13 OK13
Start13 Logstash13 Aaer13 you13 have13 wricen13 and13 verified13 the13 Logstash13 configuraon13 you13 can13 start13 Logstash13 to13 process13 records13 from13 the13 Bluemix13 apps13 (via13 Cloud13 Foundry)13 Run13 the13 following13 as13 root13 user13
[rootelkforcase]$ cd homelogstashlogstash-233
[rootelkforcase]$ binlogstash agent -f configloggregatorconf
Logstash13 is13 now13 started13
Monitor13 Logstash13 processing13 Once13 Logstash13 is13 running13 and13 log13 records13 are13 being13 produced13 by13 the13 Bluemix13 apps13 you13 can13 monitor13 progress13 in13 the13 same13 window13 that13 was13 used13 to13 start13 Logstash13 Output13 is13 similar13 to13 the13 following13
13
The13 log13 records13 being13 broadcast13 from13 the13 Bluemix13 apps13 are13 now13 sent13 to13 Elascsearch13 from13 the13 Logstash13 output13 configuraon13 Records13 are13 sent13 in13 real13 me13 If13 Logstash13 is13 not13 running13 while13 log13 records13 are13 being13 produced13 then13 these13 records13 will13 not13 be13 processed13 by13 Logstash13 and13 will13 not13 be13 stored13 in13 Elascsearch13
Page of 21 71 Page of 21 71
Use13 ELK13 The13 log13 records13 from13 the13 Bluemix13 apps13 are13 now13 being13 stored13 and13 indexed13 by13 the13 Elascsearch13 component13 This13 data13 can13 be13 accessed13 via13 the13 Kibana13 applicaon13
Kibana13 Discover13 page13 Using13 a13 web13 browser13 either13 directly13 from13 a13 client13 or13 using13 the13 VNC13 client13 access13 Kibana13 using13 the13 publicIPaddress13 of13 the13 ELK13 virtual13 server13
hcpltpublicIPaddressgt560113
No13 userpassword13 credenals13 are13 required13
Kibana13 defaults13 to13 a13 search13 of13 all13 records13 ()13 in13 the13 last13 1513 minutes13 and13 returns13 the13 results13
13
More13 complex13 searches13 are13 possible13 by13 changing13 the13 me13 filter13 and13 supplying13 search13 queries13
Change13 the13 0me13 filter13 To13 change13 the13 me13 ranges13 click13 the13 clock13 symbol13 in13 the13 top-shy‐right13 of13 the13 Kibana13 Discover13 page13
13
You13 will13 see13 the13 following13 opons13
13
Page of 22 71 Page of 22 71
A13 variety13 of13 quick13 me13 ranges13 can13 be13 selected13 or13 you13 can13 switch13 to13 relave13 or13 absolute13 me13 ranges13
Quick13 me13 ranges13 allow13 switching13 between13 last13 X13 minutes13 to13 last13 Y13 hours13 or13 Z13 days13 etc13
Relave13 me13 ranges13 allow13 more13 specific13 versions13 of13 the13 Quick13 opons13 from13 specific13 minuteshoursdays13 to13 now13
13
Absolute13 me13 ranges13 specify13 the13 start13 and13 end13 mes13 for13 the13 searches13
13
Custom13 search13 queries13 You13 can13 use13 the13 text13 window13 of13 the13 Discover13 page13 to13 search13 for13 matches13 within13 the13 log13 records13 You13 can13 use13 Apache13 Lucene13 or13 Elascsearch13 query13 DSL13 syntax13
13
Page of 23 71 Page of 23 71
Saved13 searches13 You13 can13 save13 any13 custom13 search13 syntax13 and13 choice13 of13 me13 range13 for13 future13 ease13 of13 use13 Any13 relave13 or13 absolute13 me13 frame13 is13 saved13 as13 part13 of13 the13 search13 13
To13 save13 a13 search13
1 Set13 up13 and13 run13 the13 required13 syntax13 and13 me13 frame13
13
2 Click13 the13 Save13 Search13 icon13
13
3 Type13 in13 a13 name13 for13 the13 saved13 search13
Page of 24 71 Page of 24 71
13
7 To13 delete13 or13 update13 saved13 searches13 click13 13 manage13 saved13 searches13
13
Use13 the13 edit13 icons13 to13 make13 any13 changes13
Page of 26 71 Page of 26 71
13
Build13 simple13 Kibana13 histograms13 Kibana13 allows13 you13 to13 build13 charts13 and13 dashboards13 based13 on13 the13 search13 capability13 offered13 by13 the13 Discover13 page13 Saved13 searches13 can13 be13 used13 as13 the13 basis13 for13 charts13 and13 dashboards13 built13 using13 those13 charts13 To13 build13 a13 simple13 histogram13
1 Select13 the13 Visualize13 page13
13
2 Select13 Ver0cal13 Bar13 Chart13
13
3 To13 use13 a13 saved13 search13 click13 From13 a13 saved13 search13
Page of 27 71 Page of 27 71
13
4 Select13 a13 previously13 saved13 search13
13
5 Select13 to13 edit13 the13 x-shy‐axis
6 Edit13 the13 fields13 with13 informaon13 about13 the13 aggregaon13 field13 interval13 and13 custom13 label13
Page of 28 71 Page of 28 71
13
Page of 29 71 Page of 29 71
7 Click13 the13 green13 arrow13 The13 histogram13 is13 created13 and13 run13 based13 on13 the13 saved13 search
13
8 To13 save13 the13 histogram13 click13 the13 Save13 Visualiza0on13 icon13 and13 add13 a13 tle13
13
13
9 Click13 Save13
10 To13 view13 the13 visualisaon13 click13 the13 Dashboard13 page13
Page of 30 71 Page of 30 71
13
11 Click13 the13 Add13 Visualiza0on13 icon13
13
12 Select13 the13 visiualizaon13 that13 you13 just13 created13
13
Page of 31 71 Page of 31 71
13
13 To13 save13 the13 dashboard13 click13 the13 Save13 Dashboard13 icon13
Page of 32 71 Page of 32 71
13
14 Add13 a13 tle13 and13 select13 whether13 to13 store13 the13 selected13 me13 frame13 with13 the13 dashboard13
13
15 Click13 Save13
16 You13 can13 edit13 the13 shape13 and13 number13 of13 charts13 and13 save13 the13 dashboard13
17 To13 re-shy‐run13 the13 dashboard13 select13 the13 Dashboard13 page13 and13 click13 the13 Load13 Saved13 Dashboard13 icon13
Page of 33 71 Page of 33 71
13
18 Select13 the13 saved13 dashboard13 from13 the13 list13
13
19 The13 dashboard13 runs13
13
Page of 34 71 Page of 34 71
Page of 35 71 Page of 35 71
Aler0ng13 with13 ELK13 ELK13 examines13 the13 log13 records13 in13 process13 to13 send13 alerts13 to13 third-shy‐party13 alerng13 event13 management13 and13 social13 media13 applicaons13 As13 shown13 previously13 in13 the13 ELK13 architecture13 the13 ELK13 strategy13 for13 alerng13 is13 to13 send13 alerts13 from13 the13 Logstash13 component13 directly13 Therefore13 log13 records13 and13 alerts13 are13 sent13 to13 both13 Elascsearch13 and13 third-shy‐party13 tools13 at13 the13 same13 instant13 using13 the13 output13 secon13 of13 the13 Logstash13 configuraon13 Any13 alerts13 are13 generated13 during13 the13 filter13 secon13 processing13 of13 the13 Logstash13 configuraon13
Create13 alerts13 in13 Logstash13 The13 Logstash13 filter13 secon13 is13 used13 to13 trigger13 alerts13 based13 on13 the13 log13 records13 being13 processed13 For13 example13 log13 records13 can13 be13 checked13 for13 errors13 warnings13 crashed13 or13 stopped13 messages13 and13 generate13 an13 alert13
Here13 is13 an13 example13 Logstash13 filter13 code13
13 if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
The13 specific13 tags13 are13 set13 against13 the13 log13 records13 to13 trigger13 the13 alert13 later13 in13 the13 configuraon13 in13 the13 output13 secon13
Control13 the13 number13 of13 generated13 alerts13 using13 the13 Logstash13 throcle13 plugin13 so13 that13 either13 a13 lot13 of13 instances13 in13 a13 fixed13 me13 period13 are13 required13 before13 an13 alert13 is13 sent13 or13 a13 specific13 number13 of13 instances13 triggers13 the13 alert13
Page of 36 71 Page of 36 71
Edit13 the13 plugin13 parameters13 to13 meet13 system13 requirements13
13 13 13 13 13 if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 10
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle13
after_count =gt 20
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 30
period =gt 1200
key =gt servicename_alarm_warning
Page of 37 71 Page of 37 71
add_tag =gt servicename_throttled
13 13 13 13 13 13 13 13 13
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 40
period =gt 1500
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
Enrich13 alerts13 with13 message13 detail13 Some13 of13 the13 third-shy‐party13 integraons13 benefit13 from13 having13 components13 of13 the13 raw13 log13 record13 sent13 in13 addion13 to13 the13 raw13 record13 or13 fields13 in13 the13 record13 edited13 transformed13 or13 translated13
Grok13 Logstash13 allows13 the13 log13 record13 to13 be13 broken13 up13 into13 logical13 secons13 and13 each13 secon13 stored13 in13 a13 new13 parameter13 using13 the13 grok13 parameter13 in13 the13 filter13 secon13 Here13 is13 an13 example13 grok13
13 grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
To13 build13 the13 grok13 statement13 Logstash13 offers13 an13 online13 tool13 at13 hcpsgrokdebugherokuappcom13
Translate13 If13 specific13 fields13 in13 the13 log13 record13 are13 coded13 (for13 example13 the13 Bluemix13 app13 name13 is13 oaen13 not13 sent13 in13 the13 syslog13 messages13 instead13 the13 GUID13 is13 sent13 which13 is13 not13 useful)13 then13 the13 filter13 secon13 translate13 plugin13 can13 be13 used13 to13 add13 new13 fields13 by13 holding13 the13 decoded13 versions13 of13 those13 values13
13 translate
field =gt data1
destination =gt data1
override =gt true
Page of 38 71 Page of 38 71
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
dc50f6e4-c052-436a-bf27-95ba392c4228 monverifyservicemanagement
0c677e82-5bfc-4e38-9155-e416249e3ee6 ms-catalog-postpyretic-pomiculture
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
However13 the13 translate13 plugin13 is13 not13 included13 with13 the13 Logstash13 installaon13 In13 the13 next13 secon13 I13 will13 13 describe13 how13 to13 add13 new13 plugins13 to13 the13 Logstash13 instance13
Add13 custom13 plugins13 to13 Logstash13 As13 an13 open13 source13 tool13 Logstash13 encourages13 users13 to13 develop13 content13 for13 all13 users13 to13 share13 Some13 of13 the13 plugins13 that13 were13 created13 by13 our13 community13 include13 a13 translaon13 plugin13 an13 output13 Slack13 plugin13 and13 an13 output13 syslog13 plugin13 Letrsquos13 look13 at13 how13 to13 add13 these13 plugins13 to13 your13 environment13
Filter13 translate13 plugin13 To13 add13 the13 translate13 plugin13
1 Download13 the13 plugin13
2 Copy13 the13 plugin13 file13 logstash-shy‐filter-shy‐translate-shy‐214gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-filter-translate 214
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Output13 Slack13 plugin13 To13 add13 the13 Slack13 plugin13
Page of 39 71 Page of 39 71
1 Download13 the13 plugin13 13
2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐slack-shy‐014gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-output-slack 014
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Output13 syslog13 plugin13 To13 add13 the13 syslog13 plugin13
1 Download13 the13 plugin13 13
2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐syslog-shy‐214gem13 to13 the13 ELK13 Virtual13 Server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-output-syslog 214
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Integrate13 Logstash13 with13 Slack13 To13 send13 generated13 alerts13 to13 Slack13 complete13 these13 steps13
1 Create13 a13 Slack13 integraon13 13
2 Use13 the13 Slack13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 Slack13 integraon13
113 Login13 to13 Slack13 and13 go13 to13 the13 Apps13 page13 13
Page of 40 71 Page of 40 71
13
313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13
13
A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13
13
Page of 42 71 Page of 42 71
The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13
13
The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13
13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
slack
url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5
channel =gt cloud-operations-mon
username =gt nick_cawood
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13
Page of 43 71 Page of 43 71
Page of 44 71 Page of 44 71
In13 Slack13 the13 alerts13 look13 like13 this13
13
Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13
1 Create13 a13 PagerDuty13 integraon13 13
2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13
1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13
13
2 Click13 Add13 New13 Service13
13
313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13
Page of 45 71 Page of 45 71
13
413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13
513 Click13 Add13 Service13
13
The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13
Page of 46 71 Page of 46 71
13
The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt be7eb1ea9c224018923e818d3e891d0b
incident_key =gt logstashhosttype
Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13
Page of 47 71 Page of 47 71
13
Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13
1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13
2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13
3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13
4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13
13 13 http
url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo
http_method =gt post
format =gt json
For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13
13 13 syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
Page of 48 71 Page of 48 71
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13
13
The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13
if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2
This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13
Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13
Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13
Logstash13 severity Shown13 in13 Omnibus13 as
alert Indeterminate
crical Warning
debug 7
emergency Clear
error Minor
informaonal 6
noce Crical
warning Major
Page of 49 71 Page of 49 71
13
Page of 50 71 Page of 50 71
Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13
input
tcp
port =gt 5000
type =gt syslog
filter
mutate
add_field =gt format =gt cf
grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
translate
field =gt data1
destination =gt data1
override =gt true
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
=== Check Alarm Criteria ===
Standard Key Words
if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
=== Alarm Exclusions ===
if [message] =~ (i)sometexthere
mutate remove_tag =gt servicename_alarm
Throttle alarms so we do not hit PD rate limits
Page of 52 71 Page of 52 71
if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 5
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle
after_count =gt 5
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 5
period =gt 1200
key =gt servicename_alarm_warning
add_tag =gt servicename_throttled
Page of 53 71 Page of 53 71
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
output
elasticsearch
hosts =gt localhost
stdout codec =gt rubydebug
if servicename_alarm_err in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
Page of 54 71 Page of 54 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 55 71 Page of 55 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
Page of 56 71 Page of 56 71
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert err message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_error in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
Page of 57 71 Page of 57 71
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 58 71 Page of 58 71
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 59 71 Page of 59 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert error message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
Page of 60 71 Page of 60 71
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
Page of 61 71 Page of 61 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 62 71 Page of 62 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt x
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert fatal message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
Page of 63 71 Page of 63 71
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
Page of 64 71 Page of 64 71
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 65 71 Page of 65 71
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt heavy_exclamation_mark
Page of 66 71 Page of 66 71
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert WARNING message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 67 71 Page of 67 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
Page of 68 71 Page of 68 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 69 71 Page of 69 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
Page of 70 71 Page of 70 71
Page of 71 71 Page of 71 71
elasticsearchusername user
elasticsearchpassword pass
SSL for outgoing requests from the Kibana Server to the browser (PEM formatted)
serversslcert pathtoyourservercrt
serversslkey pathtoyourserverkey
Optional setting to validate that your Elasticsearch backend uses the same key files (PEM formatted)
elasticsearchsslcert pathtoyourclientcrt
elasticsearchsslkey pathtoyourclientkey
If you need to provide a CA certificate for your Elasticsearch instance put
the path of the pem file here
elasticsearchsslca pathtoyourCApem
Set to false to have a complete disregard for the validity of the SSL
certificate
elasticsearchsslverify true
Time in milliseconds to wait for elasticsearch to respond to pings defaults to
request_timeout setting
elasticsearchpingTimeout 1500
Time in milliseconds to wait for responses from the back end or elasticsearch
This must be gt 0
elasticsearchrequestTimeout 30000
Time in milliseconds for Elasticsearch to wait for responses from shards
Set to 0 to disable
elasticsearchshardTimeout 0
Time in milliseconds to wait for Elasticsearch at Kibana startup before retrying
elasticsearchstartupTimeout 5000
elasticsearchstartupTimeout 30000
Page of 17 71 Page of 17 71
Set the path to where you would like the process id file to be created
pidfile varrunkibanapid
If you would like to send the log output to a file you can set the path below
loggingdest stdout
Set this to true to suppress all logging output
loggingsilent false
Set this to true to suppress all logging output except for error messages
loggingquiet false
Set this to true to log all events including system usage information and all requests
loggingverbose false
Kibana13 is13 now13 installed13
Start13 Kibana13 To13 start13 Kibana13 go13 to13 the13 bin13 directory13 and13 run13 the13 kibana13 executable13
[rootelkforcase]$
[rootelkforcase]$ cd homekibanakibana-451-linux-x64bin
[rootelkforcase]$ kibana amp
[1] 22283
[rootelkforcase] log [095900542] [info][status][pluginkibana] Status changed from uninitialized to green - Ready
log [095900600] [info][status][pluginelasticsearch] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [095900689] [info][status][pluginkbn_vislib_vis_types] Status changed from uninitialized to green - Ready
log [095900717] [info][status][pluginmarkdown_vis] Status changed from uninitialized to green - Ready
log [095900721] [info][status][pluginmetric_vis] Status changed from uninitialized to green - Ready
log [095900814] [info][status][pluginspyModes] Status changed from uninitialized to green - Ready
log [095900934] [info][status][pluginstatusPage] Status changed from uninitialized to green - Ready
Page of 18 71 Page of 18 71
log [095900941] [info][status][plugintable_vis] Status changed from uninitialized to green - Ready
log [095901067] [info][listening] Server running at http00005601
log [095906083] [info][status][pluginelasticsearch] Status changed from yellow to yellow - No existing Kibana index found
log [095908942] [info][status][pluginelasticsearch] Status changed from yellow to green - Kibana index ready
Kibana13 is13 now13 started13
Broadcast13 Bluemix13 app13 log13 records13 Bluemix13 apps13 (which13 are13 shown13 in13 the13 US13 South13 Bluemix13 region13 of13 the13 System13 Architecture13 diagram)13 will13 produce13 logs13 in13 a13 syslog13 rfc542413 format13 using13 a13 Bluemix13 tool13 called13 Loggregator13
For13 Logstash13 to13 receive13 these13 logs13 each13 Bluemix13 app13 needs13 to13 broadcast13 these13 logs13 To13 do13 that13 run13 the13 following13 commands13 on13 each13 Bluemix13 app13 using13 the13 public13 IP13 address13 of13 the13 ELK13 virtual13 server13 the13 ELK13 tag13 to13 trace13 the13 process13 and13 insert13 the13 name13 of13 each13 app13 in13 turn13
cf cups elk -l syslogltpublicIPaddress5000 cf bind-service ltmyappgt elk
cf restage ltmyappgt
Install13 Logstash13 To13 install13 Logstash13 as13 root13 user13 create13 a13 directory13 for13 the13 installaon13 and13 untar13 the13 installaon13 file13 there13
13 13 [rootelkforcase]$ cd home
[rootelkforcase]$ mkdir logstash
[rootelkforcase]$ cp homeibmcloudlogstash-233targz homelogstash
[rootelkforcase]$ cd logstash
[rootelkforcase]$ gunzip logstash-233targz
[rootelkforcase]$ untar logstash-233tar
Write13 Logstash13 configura0on13 Logstash13 requires13 a13 configuraon13 file13 to13 be13 wricen13 to13 receive13 transform13 and13 output13 the13 log13 records13 to13 Elascsearch13 This13 secon13 explains13 the13 basic13 requirements13 of13 processing13 Bluemix13 app13 log13 records13 into13 Elascsearch13 (more13 complex13 configuraons13 are13 detailed13 later13 in13 this13 document13 for13 the13 integraons13 with13 third-shy‐party13 tools)13
Create13 a13 conf13 file13 as13 root13 user13
[rootelkforcase]$ cd homelogstashlogstash-233
[rootelkforcase]$ mkdir config
Page of 19 71 Page of 19 71
[rootelkforcase]$ cd config
[rootelkforcase]$ touch loggregatorconf
Edit13 the13 loggregator13 file13 with13 the13 following13 contents13
13 Input13 secon13 defines13 which13 log13 records13 to13 receive13
13 port13 and13 type13 reflect13 method13 selected13 for13 Cloud13 Foundry13 to13 broadcast13 the13 records13
input
tcp
port =gt 5000
type =gt syslog
13 Filter13 secon13 ndash13 transforms13 the13 data13
filter
Add a field to identify the records as Cloud Foundry based
mutate
add_field =gt format =gt cf
13 Output13 secon13 ndash13 sends13 the13 data13 to13 its13 desnaon(s)13
output
13 Use13 the13 elascsearch13 plugin13 to13 send13 the13 data13 to13 the13 elascsearch13 instance13
13 elasticsearch
hosts =gt localhost
stdout codec =gt rubydebug
13 End13 of13 Config13
Save13 the13 file13 Verify13 the13 format13 of13 the13 file13 is13 acceptable13 by13 running13 the13 following13 commands13
[rootelkforcase]$ cd homelogstashlogstash-233
[rootelkforcase]$ binlogstash agent --configtest -f configloggregatorconf
Page of 20 71 Page of 20 71
Configuraon13 is13 OK13
Start13 Logstash13 Aaer13 you13 have13 wricen13 and13 verified13 the13 Logstash13 configuraon13 you13 can13 start13 Logstash13 to13 process13 records13 from13 the13 Bluemix13 apps13 (via13 Cloud13 Foundry)13 Run13 the13 following13 as13 root13 user13
[rootelkforcase]$ cd homelogstashlogstash-233
[rootelkforcase]$ binlogstash agent -f configloggregatorconf
Logstash13 is13 now13 started13
Monitor13 Logstash13 processing13 Once13 Logstash13 is13 running13 and13 log13 records13 are13 being13 produced13 by13 the13 Bluemix13 apps13 you13 can13 monitor13 progress13 in13 the13 same13 window13 that13 was13 used13 to13 start13 Logstash13 Output13 is13 similar13 to13 the13 following13
13
The13 log13 records13 being13 broadcast13 from13 the13 Bluemix13 apps13 are13 now13 sent13 to13 Elascsearch13 from13 the13 Logstash13 output13 configuraon13 Records13 are13 sent13 in13 real13 me13 If13 Logstash13 is13 not13 running13 while13 log13 records13 are13 being13 produced13 then13 these13 records13 will13 not13 be13 processed13 by13 Logstash13 and13 will13 not13 be13 stored13 in13 Elascsearch13
Page of 21 71 Page of 21 71
Use13 ELK13 The13 log13 records13 from13 the13 Bluemix13 apps13 are13 now13 being13 stored13 and13 indexed13 by13 the13 Elascsearch13 component13 This13 data13 can13 be13 accessed13 via13 the13 Kibana13 applicaon13
Kibana13 Discover13 page13 Using13 a13 web13 browser13 either13 directly13 from13 a13 client13 or13 using13 the13 VNC13 client13 access13 Kibana13 using13 the13 publicIPaddress13 of13 the13 ELK13 virtual13 server13
hcpltpublicIPaddressgt560113
No13 userpassword13 credenals13 are13 required13
Kibana13 defaults13 to13 a13 search13 of13 all13 records13 ()13 in13 the13 last13 1513 minutes13 and13 returns13 the13 results13
13
More13 complex13 searches13 are13 possible13 by13 changing13 the13 me13 filter13 and13 supplying13 search13 queries13
Change13 the13 0me13 filter13 To13 change13 the13 me13 ranges13 click13 the13 clock13 symbol13 in13 the13 top-shy‐right13 of13 the13 Kibana13 Discover13 page13
13
You13 will13 see13 the13 following13 opons13
13
Page of 22 71 Page of 22 71
A13 variety13 of13 quick13 me13 ranges13 can13 be13 selected13 or13 you13 can13 switch13 to13 relave13 or13 absolute13 me13 ranges13
Quick13 me13 ranges13 allow13 switching13 between13 last13 X13 minutes13 to13 last13 Y13 hours13 or13 Z13 days13 etc13
Relave13 me13 ranges13 allow13 more13 specific13 versions13 of13 the13 Quick13 opons13 from13 specific13 minuteshoursdays13 to13 now13
13
Absolute13 me13 ranges13 specify13 the13 start13 and13 end13 mes13 for13 the13 searches13
13
Custom13 search13 queries13 You13 can13 use13 the13 text13 window13 of13 the13 Discover13 page13 to13 search13 for13 matches13 within13 the13 log13 records13 You13 can13 use13 Apache13 Lucene13 or13 Elascsearch13 query13 DSL13 syntax13
13
Page of 23 71 Page of 23 71
Saved13 searches13 You13 can13 save13 any13 custom13 search13 syntax13 and13 choice13 of13 me13 range13 for13 future13 ease13 of13 use13 Any13 relave13 or13 absolute13 me13 frame13 is13 saved13 as13 part13 of13 the13 search13 13
To13 save13 a13 search13
1 Set13 up13 and13 run13 the13 required13 syntax13 and13 me13 frame13
13
2 Click13 the13 Save13 Search13 icon13
13
3 Type13 in13 a13 name13 for13 the13 saved13 search13
Page of 24 71 Page of 24 71
13
7 To13 delete13 or13 update13 saved13 searches13 click13 13 manage13 saved13 searches13
13
Use13 the13 edit13 icons13 to13 make13 any13 changes13
Page of 26 71 Page of 26 71
13
Build13 simple13 Kibana13 histograms13 Kibana13 allows13 you13 to13 build13 charts13 and13 dashboards13 based13 on13 the13 search13 capability13 offered13 by13 the13 Discover13 page13 Saved13 searches13 can13 be13 used13 as13 the13 basis13 for13 charts13 and13 dashboards13 built13 using13 those13 charts13 To13 build13 a13 simple13 histogram13
1 Select13 the13 Visualize13 page13
13
2 Select13 Ver0cal13 Bar13 Chart13
13
3 To13 use13 a13 saved13 search13 click13 From13 a13 saved13 search13
Page of 27 71 Page of 27 71
13
4 Select13 a13 previously13 saved13 search13
13
5 Select13 to13 edit13 the13 x-shy‐axis
6 Edit13 the13 fields13 with13 informaon13 about13 the13 aggregaon13 field13 interval13 and13 custom13 label13
Page of 28 71 Page of 28 71
13
Page of 29 71 Page of 29 71
7 Click13 the13 green13 arrow13 The13 histogram13 is13 created13 and13 run13 based13 on13 the13 saved13 search
13
8 To13 save13 the13 histogram13 click13 the13 Save13 Visualiza0on13 icon13 and13 add13 a13 tle13
13
13
9 Click13 Save13
10 To13 view13 the13 visualisaon13 click13 the13 Dashboard13 page13
Page of 30 71 Page of 30 71
13
11 Click13 the13 Add13 Visualiza0on13 icon13
13
12 Select13 the13 visiualizaon13 that13 you13 just13 created13
13
Page of 31 71 Page of 31 71
13
13 To13 save13 the13 dashboard13 click13 the13 Save13 Dashboard13 icon13
Page of 32 71 Page of 32 71
13
14 Add13 a13 tle13 and13 select13 whether13 to13 store13 the13 selected13 me13 frame13 with13 the13 dashboard13
13
15 Click13 Save13
16 You13 can13 edit13 the13 shape13 and13 number13 of13 charts13 and13 save13 the13 dashboard13
17 To13 re-shy‐run13 the13 dashboard13 select13 the13 Dashboard13 page13 and13 click13 the13 Load13 Saved13 Dashboard13 icon13
Page of 33 71 Page of 33 71
13
18 Select13 the13 saved13 dashboard13 from13 the13 list13
13
19 The13 dashboard13 runs13
13
Page of 34 71 Page of 34 71
Page of 35 71 Page of 35 71
Aler0ng13 with13 ELK13 ELK13 examines13 the13 log13 records13 in13 process13 to13 send13 alerts13 to13 third-shy‐party13 alerng13 event13 management13 and13 social13 media13 applicaons13 As13 shown13 previously13 in13 the13 ELK13 architecture13 the13 ELK13 strategy13 for13 alerng13 is13 to13 send13 alerts13 from13 the13 Logstash13 component13 directly13 Therefore13 log13 records13 and13 alerts13 are13 sent13 to13 both13 Elascsearch13 and13 third-shy‐party13 tools13 at13 the13 same13 instant13 using13 the13 output13 secon13 of13 the13 Logstash13 configuraon13 Any13 alerts13 are13 generated13 during13 the13 filter13 secon13 processing13 of13 the13 Logstash13 configuraon13
Create13 alerts13 in13 Logstash13 The13 Logstash13 filter13 secon13 is13 used13 to13 trigger13 alerts13 based13 on13 the13 log13 records13 being13 processed13 For13 example13 log13 records13 can13 be13 checked13 for13 errors13 warnings13 crashed13 or13 stopped13 messages13 and13 generate13 an13 alert13
Here13 is13 an13 example13 Logstash13 filter13 code13
13 if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
The13 specific13 tags13 are13 set13 against13 the13 log13 records13 to13 trigger13 the13 alert13 later13 in13 the13 configuraon13 in13 the13 output13 secon13
Control13 the13 number13 of13 generated13 alerts13 using13 the13 Logstash13 throcle13 plugin13 so13 that13 either13 a13 lot13 of13 instances13 in13 a13 fixed13 me13 period13 are13 required13 before13 an13 alert13 is13 sent13 or13 a13 specific13 number13 of13 instances13 triggers13 the13 alert13
Page of 36 71 Page of 36 71
Edit13 the13 plugin13 parameters13 to13 meet13 system13 requirements13
13 13 13 13 13 if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 10
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle13
after_count =gt 20
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 30
period =gt 1200
key =gt servicename_alarm_warning
Page of 37 71 Page of 37 71
add_tag =gt servicename_throttled
13 13 13 13 13 13 13 13 13
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 40
period =gt 1500
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
Enrich13 alerts13 with13 message13 detail13 Some13 of13 the13 third-shy‐party13 integraons13 benefit13 from13 having13 components13 of13 the13 raw13 log13 record13 sent13 in13 addion13 to13 the13 raw13 record13 or13 fields13 in13 the13 record13 edited13 transformed13 or13 translated13
Grok13 Logstash13 allows13 the13 log13 record13 to13 be13 broken13 up13 into13 logical13 secons13 and13 each13 secon13 stored13 in13 a13 new13 parameter13 using13 the13 grok13 parameter13 in13 the13 filter13 secon13 Here13 is13 an13 example13 grok13
13 grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
To13 build13 the13 grok13 statement13 Logstash13 offers13 an13 online13 tool13 at13 hcpsgrokdebugherokuappcom13
Translate13 If13 specific13 fields13 in13 the13 log13 record13 are13 coded13 (for13 example13 the13 Bluemix13 app13 name13 is13 oaen13 not13 sent13 in13 the13 syslog13 messages13 instead13 the13 GUID13 is13 sent13 which13 is13 not13 useful)13 then13 the13 filter13 secon13 translate13 plugin13 can13 be13 used13 to13 add13 new13 fields13 by13 holding13 the13 decoded13 versions13 of13 those13 values13
13 translate
field =gt data1
destination =gt data1
override =gt true
Page of 38 71 Page of 38 71
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
dc50f6e4-c052-436a-bf27-95ba392c4228 monverifyservicemanagement
0c677e82-5bfc-4e38-9155-e416249e3ee6 ms-catalog-postpyretic-pomiculture
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
However13 the13 translate13 plugin13 is13 not13 included13 with13 the13 Logstash13 installaon13 In13 the13 next13 secon13 I13 will13 13 describe13 how13 to13 add13 new13 plugins13 to13 the13 Logstash13 instance13
Add13 custom13 plugins13 to13 Logstash13 As13 an13 open13 source13 tool13 Logstash13 encourages13 users13 to13 develop13 content13 for13 all13 users13 to13 share13 Some13 of13 the13 plugins13 that13 were13 created13 by13 our13 community13 include13 a13 translaon13 plugin13 an13 output13 Slack13 plugin13 and13 an13 output13 syslog13 plugin13 Letrsquos13 look13 at13 how13 to13 add13 these13 plugins13 to13 your13 environment13
Filter13 translate13 plugin13 To13 add13 the13 translate13 plugin13
1 Download13 the13 plugin13
2 Copy13 the13 plugin13 file13 logstash-shy‐filter-shy‐translate-shy‐214gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-filter-translate 214
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Output13 Slack13 plugin13 To13 add13 the13 Slack13 plugin13
Page of 39 71 Page of 39 71
1 Download13 the13 plugin13 13
2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐slack-shy‐014gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-output-slack 014
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Output13 syslog13 plugin13 To13 add13 the13 syslog13 plugin13
1 Download13 the13 plugin13 13
2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐syslog-shy‐214gem13 to13 the13 ELK13 Virtual13 Server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-output-syslog 214
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Integrate13 Logstash13 with13 Slack13 To13 send13 generated13 alerts13 to13 Slack13 complete13 these13 steps13
1 Create13 a13 Slack13 integraon13 13
2 Use13 the13 Slack13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 Slack13 integraon13
113 Login13 to13 Slack13 and13 go13 to13 the13 Apps13 page13 13
Page of 40 71 Page of 40 71
13
313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13
13
A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13
13
Page of 42 71 Page of 42 71
The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13
13
The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13
13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
slack
url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5
channel =gt cloud-operations-mon
username =gt nick_cawood
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13
Page of 43 71 Page of 43 71
Page of 44 71 Page of 44 71
In13 Slack13 the13 alerts13 look13 like13 this13
13
Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13
1 Create13 a13 PagerDuty13 integraon13 13
2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13
1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13
13
2 Click13 Add13 New13 Service13
13
313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13
Page of 45 71 Page of 45 71
13
413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13
513 Click13 Add13 Service13
13
The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13
Page of 46 71 Page of 46 71
13
The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt be7eb1ea9c224018923e818d3e891d0b
incident_key =gt logstashhosttype
Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13
Page of 47 71 Page of 47 71
13
Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13
1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13
2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13
3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13
4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13
13 13 http
url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo
http_method =gt post
format =gt json
For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13
13 13 syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
Page of 48 71 Page of 48 71
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13
13
The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13
if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2
This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13
Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13
Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13
Logstash13 severity Shown13 in13 Omnibus13 as
alert Indeterminate
crical Warning
debug 7
emergency Clear
error Minor
informaonal 6
noce Crical
warning Major
Page of 49 71 Page of 49 71
13
Page of 50 71 Page of 50 71
Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13
input
tcp
port =gt 5000
type =gt syslog
filter
mutate
add_field =gt format =gt cf
grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
translate
field =gt data1
destination =gt data1
override =gt true
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
=== Check Alarm Criteria ===
Standard Key Words
if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
=== Alarm Exclusions ===
if [message] =~ (i)sometexthere
mutate remove_tag =gt servicename_alarm
Throttle alarms so we do not hit PD rate limits
Page of 52 71 Page of 52 71
if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 5
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle
after_count =gt 5
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 5
period =gt 1200
key =gt servicename_alarm_warning
add_tag =gt servicename_throttled
Page of 53 71 Page of 53 71
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
output
elasticsearch
hosts =gt localhost
stdout codec =gt rubydebug
if servicename_alarm_err in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
Page of 54 71 Page of 54 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 55 71 Page of 55 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
Page of 56 71 Page of 56 71
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert err message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_error in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
Page of 57 71 Page of 57 71
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 58 71 Page of 58 71
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 59 71 Page of 59 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert error message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
Page of 60 71 Page of 60 71
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
Page of 61 71 Page of 61 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 62 71 Page of 62 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt x
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert fatal message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
Page of 63 71 Page of 63 71
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
Page of 64 71 Page of 64 71
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 65 71 Page of 65 71
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt heavy_exclamation_mark
Page of 66 71 Page of 66 71
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert WARNING message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 67 71 Page of 67 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
Page of 68 71 Page of 68 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 69 71 Page of 69 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
Page of 70 71 Page of 70 71
Page of 71 71 Page of 71 71
Set the path to where you would like the process id file to be created
pidfile varrunkibanapid
If you would like to send the log output to a file you can set the path below
loggingdest stdout
Set this to true to suppress all logging output
loggingsilent false
Set this to true to suppress all logging output except for error messages
loggingquiet false
Set this to true to log all events including system usage information and all requests
loggingverbose false
Kibana13 is13 now13 installed13
Start13 Kibana13 To13 start13 Kibana13 go13 to13 the13 bin13 directory13 and13 run13 the13 kibana13 executable13
[rootelkforcase]$
[rootelkforcase]$ cd homekibanakibana-451-linux-x64bin
[rootelkforcase]$ kibana amp
[1] 22283
[rootelkforcase] log [095900542] [info][status][pluginkibana] Status changed from uninitialized to green - Ready
log [095900600] [info][status][pluginelasticsearch] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [095900689] [info][status][pluginkbn_vislib_vis_types] Status changed from uninitialized to green - Ready
log [095900717] [info][status][pluginmarkdown_vis] Status changed from uninitialized to green - Ready
log [095900721] [info][status][pluginmetric_vis] Status changed from uninitialized to green - Ready
log [095900814] [info][status][pluginspyModes] Status changed from uninitialized to green - Ready
log [095900934] [info][status][pluginstatusPage] Status changed from uninitialized to green - Ready
Page of 18 71 Page of 18 71
log [095900941] [info][status][plugintable_vis] Status changed from uninitialized to green - Ready
log [095901067] [info][listening] Server running at http00005601
log [095906083] [info][status][pluginelasticsearch] Status changed from yellow to yellow - No existing Kibana index found
log [095908942] [info][status][pluginelasticsearch] Status changed from yellow to green - Kibana index ready
Kibana13 is13 now13 started13
Broadcast13 Bluemix13 app13 log13 records13 Bluemix13 apps13 (which13 are13 shown13 in13 the13 US13 South13 Bluemix13 region13 of13 the13 System13 Architecture13 diagram)13 will13 produce13 logs13 in13 a13 syslog13 rfc542413 format13 using13 a13 Bluemix13 tool13 called13 Loggregator13
For13 Logstash13 to13 receive13 these13 logs13 each13 Bluemix13 app13 needs13 to13 broadcast13 these13 logs13 To13 do13 that13 run13 the13 following13 commands13 on13 each13 Bluemix13 app13 using13 the13 public13 IP13 address13 of13 the13 ELK13 virtual13 server13 the13 ELK13 tag13 to13 trace13 the13 process13 and13 insert13 the13 name13 of13 each13 app13 in13 turn13
cf cups elk -l syslogltpublicIPaddress5000 cf bind-service ltmyappgt elk
cf restage ltmyappgt
Install13 Logstash13 To13 install13 Logstash13 as13 root13 user13 create13 a13 directory13 for13 the13 installaon13 and13 untar13 the13 installaon13 file13 there13
13 13 [rootelkforcase]$ cd home
[rootelkforcase]$ mkdir logstash
[rootelkforcase]$ cp homeibmcloudlogstash-233targz homelogstash
[rootelkforcase]$ cd logstash
[rootelkforcase]$ gunzip logstash-233targz
[rootelkforcase]$ untar logstash-233tar
Write13 Logstash13 configura0on13 Logstash13 requires13 a13 configuraon13 file13 to13 be13 wricen13 to13 receive13 transform13 and13 output13 the13 log13 records13 to13 Elascsearch13 This13 secon13 explains13 the13 basic13 requirements13 of13 processing13 Bluemix13 app13 log13 records13 into13 Elascsearch13 (more13 complex13 configuraons13 are13 detailed13 later13 in13 this13 document13 for13 the13 integraons13 with13 third-shy‐party13 tools)13
Create13 a13 conf13 file13 as13 root13 user13
[rootelkforcase]$ cd homelogstashlogstash-233
[rootelkforcase]$ mkdir config
Page of 19 71 Page of 19 71
[rootelkforcase]$ cd config
[rootelkforcase]$ touch loggregatorconf
Edit13 the13 loggregator13 file13 with13 the13 following13 contents13
13 Input13 secon13 defines13 which13 log13 records13 to13 receive13
13 port13 and13 type13 reflect13 method13 selected13 for13 Cloud13 Foundry13 to13 broadcast13 the13 records13
input
tcp
port =gt 5000
type =gt syslog
13 Filter13 secon13 ndash13 transforms13 the13 data13
filter
Add a field to identify the records as Cloud Foundry based
mutate
add_field =gt format =gt cf
13 Output13 secon13 ndash13 sends13 the13 data13 to13 its13 desnaon(s)13
output
13 Use13 the13 elascsearch13 plugin13 to13 send13 the13 data13 to13 the13 elascsearch13 instance13
13 elasticsearch
hosts =gt localhost
stdout codec =gt rubydebug
13 End13 of13 Config13
Save13 the13 file13 Verify13 the13 format13 of13 the13 file13 is13 acceptable13 by13 running13 the13 following13 commands13
[rootelkforcase]$ cd homelogstashlogstash-233
[rootelkforcase]$ binlogstash agent --configtest -f configloggregatorconf
Page of 20 71 Page of 20 71
Configuraon13 is13 OK13
Start13 Logstash13 Aaer13 you13 have13 wricen13 and13 verified13 the13 Logstash13 configuraon13 you13 can13 start13 Logstash13 to13 process13 records13 from13 the13 Bluemix13 apps13 (via13 Cloud13 Foundry)13 Run13 the13 following13 as13 root13 user13
[rootelkforcase]$ cd homelogstashlogstash-233
[rootelkforcase]$ binlogstash agent -f configloggregatorconf
Logstash13 is13 now13 started13
Monitor13 Logstash13 processing13 Once13 Logstash13 is13 running13 and13 log13 records13 are13 being13 produced13 by13 the13 Bluemix13 apps13 you13 can13 monitor13 progress13 in13 the13 same13 window13 that13 was13 used13 to13 start13 Logstash13 Output13 is13 similar13 to13 the13 following13
13
The13 log13 records13 being13 broadcast13 from13 the13 Bluemix13 apps13 are13 now13 sent13 to13 Elascsearch13 from13 the13 Logstash13 output13 configuraon13 Records13 are13 sent13 in13 real13 me13 If13 Logstash13 is13 not13 running13 while13 log13 records13 are13 being13 produced13 then13 these13 records13 will13 not13 be13 processed13 by13 Logstash13 and13 will13 not13 be13 stored13 in13 Elascsearch13
Page of 21 71 Page of 21 71
Use13 ELK13 The13 log13 records13 from13 the13 Bluemix13 apps13 are13 now13 being13 stored13 and13 indexed13 by13 the13 Elascsearch13 component13 This13 data13 can13 be13 accessed13 via13 the13 Kibana13 applicaon13
Kibana13 Discover13 page13 Using13 a13 web13 browser13 either13 directly13 from13 a13 client13 or13 using13 the13 VNC13 client13 access13 Kibana13 using13 the13 publicIPaddress13 of13 the13 ELK13 virtual13 server13
hcpltpublicIPaddressgt560113
No13 userpassword13 credenals13 are13 required13
Kibana13 defaults13 to13 a13 search13 of13 all13 records13 ()13 in13 the13 last13 1513 minutes13 and13 returns13 the13 results13
13
More13 complex13 searches13 are13 possible13 by13 changing13 the13 me13 filter13 and13 supplying13 search13 queries13
Change13 the13 0me13 filter13 To13 change13 the13 me13 ranges13 click13 the13 clock13 symbol13 in13 the13 top-shy‐right13 of13 the13 Kibana13 Discover13 page13
13
You13 will13 see13 the13 following13 opons13
13
Page of 22 71 Page of 22 71
A13 variety13 of13 quick13 me13 ranges13 can13 be13 selected13 or13 you13 can13 switch13 to13 relave13 or13 absolute13 me13 ranges13
Quick13 me13 ranges13 allow13 switching13 between13 last13 X13 minutes13 to13 last13 Y13 hours13 or13 Z13 days13 etc13
Relave13 me13 ranges13 allow13 more13 specific13 versions13 of13 the13 Quick13 opons13 from13 specific13 minuteshoursdays13 to13 now13
13
Absolute13 me13 ranges13 specify13 the13 start13 and13 end13 mes13 for13 the13 searches13
13
Custom13 search13 queries13 You13 can13 use13 the13 text13 window13 of13 the13 Discover13 page13 to13 search13 for13 matches13 within13 the13 log13 records13 You13 can13 use13 Apache13 Lucene13 or13 Elascsearch13 query13 DSL13 syntax13
13
Page of 23 71 Page of 23 71
Saved13 searches13 You13 can13 save13 any13 custom13 search13 syntax13 and13 choice13 of13 me13 range13 for13 future13 ease13 of13 use13 Any13 relave13 or13 absolute13 me13 frame13 is13 saved13 as13 part13 of13 the13 search13 13
To13 save13 a13 search13
1 Set13 up13 and13 run13 the13 required13 syntax13 and13 me13 frame13
13
2 Click13 the13 Save13 Search13 icon13
13
3 Type13 in13 a13 name13 for13 the13 saved13 search13
Page of 24 71 Page of 24 71
13
7 To13 delete13 or13 update13 saved13 searches13 click13 13 manage13 saved13 searches13
13
Use13 the13 edit13 icons13 to13 make13 any13 changes13
Page of 26 71 Page of 26 71
13
Build13 simple13 Kibana13 histograms13 Kibana13 allows13 you13 to13 build13 charts13 and13 dashboards13 based13 on13 the13 search13 capability13 offered13 by13 the13 Discover13 page13 Saved13 searches13 can13 be13 used13 as13 the13 basis13 for13 charts13 and13 dashboards13 built13 using13 those13 charts13 To13 build13 a13 simple13 histogram13
1 Select13 the13 Visualize13 page13
13
2 Select13 Ver0cal13 Bar13 Chart13
13
3 To13 use13 a13 saved13 search13 click13 From13 a13 saved13 search13
Page of 27 71 Page of 27 71
13
4 Select13 a13 previously13 saved13 search13
13
5 Select13 to13 edit13 the13 x-shy‐axis
6 Edit13 the13 fields13 with13 informaon13 about13 the13 aggregaon13 field13 interval13 and13 custom13 label13
Page of 28 71 Page of 28 71
13
Page of 29 71 Page of 29 71
7 Click13 the13 green13 arrow13 The13 histogram13 is13 created13 and13 run13 based13 on13 the13 saved13 search
13
8 To13 save13 the13 histogram13 click13 the13 Save13 Visualiza0on13 icon13 and13 add13 a13 tle13
13
13
9 Click13 Save13
10 To13 view13 the13 visualisaon13 click13 the13 Dashboard13 page13
Page of 30 71 Page of 30 71
13
11 Click13 the13 Add13 Visualiza0on13 icon13
13
12 Select13 the13 visiualizaon13 that13 you13 just13 created13
13
Page of 31 71 Page of 31 71
13
13 To13 save13 the13 dashboard13 click13 the13 Save13 Dashboard13 icon13
Page of 32 71 Page of 32 71
13
14 Add13 a13 tle13 and13 select13 whether13 to13 store13 the13 selected13 me13 frame13 with13 the13 dashboard13
13
15 Click13 Save13
16 You13 can13 edit13 the13 shape13 and13 number13 of13 charts13 and13 save13 the13 dashboard13
17 To13 re-shy‐run13 the13 dashboard13 select13 the13 Dashboard13 page13 and13 click13 the13 Load13 Saved13 Dashboard13 icon13
Page of 33 71 Page of 33 71
13
18 Select13 the13 saved13 dashboard13 from13 the13 list13
13
19 The13 dashboard13 runs13
13
Page of 34 71 Page of 34 71
Page of 35 71 Page of 35 71
Aler0ng13 with13 ELK13 ELK13 examines13 the13 log13 records13 in13 process13 to13 send13 alerts13 to13 third-shy‐party13 alerng13 event13 management13 and13 social13 media13 applicaons13 As13 shown13 previously13 in13 the13 ELK13 architecture13 the13 ELK13 strategy13 for13 alerng13 is13 to13 send13 alerts13 from13 the13 Logstash13 component13 directly13 Therefore13 log13 records13 and13 alerts13 are13 sent13 to13 both13 Elascsearch13 and13 third-shy‐party13 tools13 at13 the13 same13 instant13 using13 the13 output13 secon13 of13 the13 Logstash13 configuraon13 Any13 alerts13 are13 generated13 during13 the13 filter13 secon13 processing13 of13 the13 Logstash13 configuraon13
Create13 alerts13 in13 Logstash13 The13 Logstash13 filter13 secon13 is13 used13 to13 trigger13 alerts13 based13 on13 the13 log13 records13 being13 processed13 For13 example13 log13 records13 can13 be13 checked13 for13 errors13 warnings13 crashed13 or13 stopped13 messages13 and13 generate13 an13 alert13
Here13 is13 an13 example13 Logstash13 filter13 code13
13 if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
The13 specific13 tags13 are13 set13 against13 the13 log13 records13 to13 trigger13 the13 alert13 later13 in13 the13 configuraon13 in13 the13 output13 secon13
Control13 the13 number13 of13 generated13 alerts13 using13 the13 Logstash13 throcle13 plugin13 so13 that13 either13 a13 lot13 of13 instances13 in13 a13 fixed13 me13 period13 are13 required13 before13 an13 alert13 is13 sent13 or13 a13 specific13 number13 of13 instances13 triggers13 the13 alert13
Page of 36 71 Page of 36 71
Edit13 the13 plugin13 parameters13 to13 meet13 system13 requirements13
13 13 13 13 13 if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 10
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle13
after_count =gt 20
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 30
period =gt 1200
key =gt servicename_alarm_warning
Page of 37 71 Page of 37 71
add_tag =gt servicename_throttled
13 13 13 13 13 13 13 13 13
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 40
period =gt 1500
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
Enrich13 alerts13 with13 message13 detail13 Some13 of13 the13 third-shy‐party13 integraons13 benefit13 from13 having13 components13 of13 the13 raw13 log13 record13 sent13 in13 addion13 to13 the13 raw13 record13 or13 fields13 in13 the13 record13 edited13 transformed13 or13 translated13
Grok13 Logstash13 allows13 the13 log13 record13 to13 be13 broken13 up13 into13 logical13 secons13 and13 each13 secon13 stored13 in13 a13 new13 parameter13 using13 the13 grok13 parameter13 in13 the13 filter13 secon13 Here13 is13 an13 example13 grok13
13 grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
To13 build13 the13 grok13 statement13 Logstash13 offers13 an13 online13 tool13 at13 hcpsgrokdebugherokuappcom13
Translate13 If13 specific13 fields13 in13 the13 log13 record13 are13 coded13 (for13 example13 the13 Bluemix13 app13 name13 is13 oaen13 not13 sent13 in13 the13 syslog13 messages13 instead13 the13 GUID13 is13 sent13 which13 is13 not13 useful)13 then13 the13 filter13 secon13 translate13 plugin13 can13 be13 used13 to13 add13 new13 fields13 by13 holding13 the13 decoded13 versions13 of13 those13 values13
13 translate
field =gt data1
destination =gt data1
override =gt true
Page of 38 71 Page of 38 71
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
dc50f6e4-c052-436a-bf27-95ba392c4228 monverifyservicemanagement
0c677e82-5bfc-4e38-9155-e416249e3ee6 ms-catalog-postpyretic-pomiculture
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
However13 the13 translate13 plugin13 is13 not13 included13 with13 the13 Logstash13 installaon13 In13 the13 next13 secon13 I13 will13 13 describe13 how13 to13 add13 new13 plugins13 to13 the13 Logstash13 instance13
Add13 custom13 plugins13 to13 Logstash13 As13 an13 open13 source13 tool13 Logstash13 encourages13 users13 to13 develop13 content13 for13 all13 users13 to13 share13 Some13 of13 the13 plugins13 that13 were13 created13 by13 our13 community13 include13 a13 translaon13 plugin13 an13 output13 Slack13 plugin13 and13 an13 output13 syslog13 plugin13 Letrsquos13 look13 at13 how13 to13 add13 these13 plugins13 to13 your13 environment13
Filter13 translate13 plugin13 To13 add13 the13 translate13 plugin13
1 Download13 the13 plugin13
2 Copy13 the13 plugin13 file13 logstash-shy‐filter-shy‐translate-shy‐214gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-filter-translate 214
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Output13 Slack13 plugin13 To13 add13 the13 Slack13 plugin13
Page of 39 71 Page of 39 71
1 Download13 the13 plugin13 13
2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐slack-shy‐014gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-output-slack 014
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Output13 syslog13 plugin13 To13 add13 the13 syslog13 plugin13
1 Download13 the13 plugin13 13
2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐syslog-shy‐214gem13 to13 the13 ELK13 Virtual13 Server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-output-syslog 214
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Integrate13 Logstash13 with13 Slack13 To13 send13 generated13 alerts13 to13 Slack13 complete13 these13 steps13
1 Create13 a13 Slack13 integraon13 13
2 Use13 the13 Slack13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 Slack13 integraon13
113 Login13 to13 Slack13 and13 go13 to13 the13 Apps13 page13 13
Page of 40 71 Page of 40 71
13
313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13
13
A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13
13
Page of 42 71 Page of 42 71
The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13
13
The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13
13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
slack
url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5
channel =gt cloud-operations-mon
username =gt nick_cawood
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13
Page of 43 71 Page of 43 71
Page of 44 71 Page of 44 71
In13 Slack13 the13 alerts13 look13 like13 this13
13
Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13
1 Create13 a13 PagerDuty13 integraon13 13
2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13
1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13
13
2 Click13 Add13 New13 Service13
13
313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13
Page of 45 71 Page of 45 71
13
413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13
513 Click13 Add13 Service13
13
The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13
Page of 46 71 Page of 46 71
13
The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt be7eb1ea9c224018923e818d3e891d0b
incident_key =gt logstashhosttype
Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13
Page of 47 71 Page of 47 71
13
Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13
1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13
2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13
3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13
4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13
13 13 http
url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo
http_method =gt post
format =gt json
For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13
13 13 syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
Page of 48 71 Page of 48 71
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13
13
The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13
if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2
This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13
Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13
Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13
Logstash13 severity Shown13 in13 Omnibus13 as
alert Indeterminate
crical Warning
debug 7
emergency Clear
error Minor
informaonal 6
noce Crical
warning Major
Page of 49 71 Page of 49 71
13
Page of 50 71 Page of 50 71
Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13
input
tcp
port =gt 5000
type =gt syslog
filter
mutate
add_field =gt format =gt cf
grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
translate
field =gt data1
destination =gt data1
override =gt true
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
=== Check Alarm Criteria ===
Standard Key Words
if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
=== Alarm Exclusions ===
if [message] =~ (i)sometexthere
mutate remove_tag =gt servicename_alarm
Throttle alarms so we do not hit PD rate limits
Page of 52 71 Page of 52 71
if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 5
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle
after_count =gt 5
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 5
period =gt 1200
key =gt servicename_alarm_warning
add_tag =gt servicename_throttled
Page of 53 71 Page of 53 71
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
output
elasticsearch
hosts =gt localhost
stdout codec =gt rubydebug
if servicename_alarm_err in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
Page of 54 71 Page of 54 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 55 71 Page of 55 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
Page of 56 71 Page of 56 71
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert err message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_error in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
Page of 57 71 Page of 57 71
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 58 71 Page of 58 71
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 59 71 Page of 59 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert error message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
Page of 60 71 Page of 60 71
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
Page of 61 71 Page of 61 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 62 71 Page of 62 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt x
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert fatal message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
Page of 63 71 Page of 63 71
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
Page of 64 71 Page of 64 71
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 65 71 Page of 65 71
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt heavy_exclamation_mark
Page of 66 71 Page of 66 71
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert WARNING message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 67 71 Page of 67 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
Page of 68 71 Page of 68 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 69 71 Page of 69 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
Page of 70 71 Page of 70 71
Page of 71 71 Page of 71 71
log [095900941] [info][status][plugintable_vis] Status changed from uninitialized to green - Ready
log [095901067] [info][listening] Server running at http00005601
log [095906083] [info][status][pluginelasticsearch] Status changed from yellow to yellow - No existing Kibana index found
log [095908942] [info][status][pluginelasticsearch] Status changed from yellow to green - Kibana index ready
Kibana13 is13 now13 started13
Broadcast13 Bluemix13 app13 log13 records13 Bluemix13 apps13 (which13 are13 shown13 in13 the13 US13 South13 Bluemix13 region13 of13 the13 System13 Architecture13 diagram)13 will13 produce13 logs13 in13 a13 syslog13 rfc542413 format13 using13 a13 Bluemix13 tool13 called13 Loggregator13
For13 Logstash13 to13 receive13 these13 logs13 each13 Bluemix13 app13 needs13 to13 broadcast13 these13 logs13 To13 do13 that13 run13 the13 following13 commands13 on13 each13 Bluemix13 app13 using13 the13 public13 IP13 address13 of13 the13 ELK13 virtual13 server13 the13 ELK13 tag13 to13 trace13 the13 process13 and13 insert13 the13 name13 of13 each13 app13 in13 turn13
cf cups elk -l syslogltpublicIPaddress5000 cf bind-service ltmyappgt elk
cf restage ltmyappgt
Install13 Logstash13 To13 install13 Logstash13 as13 root13 user13 create13 a13 directory13 for13 the13 installaon13 and13 untar13 the13 installaon13 file13 there13
13 13 [rootelkforcase]$ cd home
[rootelkforcase]$ mkdir logstash
[rootelkforcase]$ cp homeibmcloudlogstash-233targz homelogstash
[rootelkforcase]$ cd logstash
[rootelkforcase]$ gunzip logstash-233targz
[rootelkforcase]$ untar logstash-233tar
Write13 Logstash13 configura0on13 Logstash13 requires13 a13 configuraon13 file13 to13 be13 wricen13 to13 receive13 transform13 and13 output13 the13 log13 records13 to13 Elascsearch13 This13 secon13 explains13 the13 basic13 requirements13 of13 processing13 Bluemix13 app13 log13 records13 into13 Elascsearch13 (more13 complex13 configuraons13 are13 detailed13 later13 in13 this13 document13 for13 the13 integraons13 with13 third-shy‐party13 tools)13
Create13 a13 conf13 file13 as13 root13 user13
[rootelkforcase]$ cd homelogstashlogstash-233
[rootelkforcase]$ mkdir config
Page of 19 71 Page of 19 71
[rootelkforcase]$ cd config
[rootelkforcase]$ touch loggregatorconf
Edit13 the13 loggregator13 file13 with13 the13 following13 contents13
13 Input13 secon13 defines13 which13 log13 records13 to13 receive13
13 port13 and13 type13 reflect13 method13 selected13 for13 Cloud13 Foundry13 to13 broadcast13 the13 records13
input
tcp
port =gt 5000
type =gt syslog
13 Filter13 secon13 ndash13 transforms13 the13 data13
filter
Add a field to identify the records as Cloud Foundry based
mutate
add_field =gt format =gt cf
13 Output13 secon13 ndash13 sends13 the13 data13 to13 its13 desnaon(s)13
output
13 Use13 the13 elascsearch13 plugin13 to13 send13 the13 data13 to13 the13 elascsearch13 instance13
13 elasticsearch
hosts =gt localhost
stdout codec =gt rubydebug
13 End13 of13 Config13
Save13 the13 file13 Verify13 the13 format13 of13 the13 file13 is13 acceptable13 by13 running13 the13 following13 commands13
[rootelkforcase]$ cd homelogstashlogstash-233
[rootelkforcase]$ binlogstash agent --configtest -f configloggregatorconf
Page of 20 71 Page of 20 71
Configuraon13 is13 OK13
Start13 Logstash13 Aaer13 you13 have13 wricen13 and13 verified13 the13 Logstash13 configuraon13 you13 can13 start13 Logstash13 to13 process13 records13 from13 the13 Bluemix13 apps13 (via13 Cloud13 Foundry)13 Run13 the13 following13 as13 root13 user13
[rootelkforcase]$ cd homelogstashlogstash-233
[rootelkforcase]$ binlogstash agent -f configloggregatorconf
Logstash13 is13 now13 started13
Monitor13 Logstash13 processing13 Once13 Logstash13 is13 running13 and13 log13 records13 are13 being13 produced13 by13 the13 Bluemix13 apps13 you13 can13 monitor13 progress13 in13 the13 same13 window13 that13 was13 used13 to13 start13 Logstash13 Output13 is13 similar13 to13 the13 following13
13
The13 log13 records13 being13 broadcast13 from13 the13 Bluemix13 apps13 are13 now13 sent13 to13 Elascsearch13 from13 the13 Logstash13 output13 configuraon13 Records13 are13 sent13 in13 real13 me13 If13 Logstash13 is13 not13 running13 while13 log13 records13 are13 being13 produced13 then13 these13 records13 will13 not13 be13 processed13 by13 Logstash13 and13 will13 not13 be13 stored13 in13 Elascsearch13
Page of 21 71 Page of 21 71
Use13 ELK13 The13 log13 records13 from13 the13 Bluemix13 apps13 are13 now13 being13 stored13 and13 indexed13 by13 the13 Elascsearch13 component13 This13 data13 can13 be13 accessed13 via13 the13 Kibana13 applicaon13
Kibana13 Discover13 page13 Using13 a13 web13 browser13 either13 directly13 from13 a13 client13 or13 using13 the13 VNC13 client13 access13 Kibana13 using13 the13 publicIPaddress13 of13 the13 ELK13 virtual13 server13
hcpltpublicIPaddressgt560113
No13 userpassword13 credenals13 are13 required13
Kibana13 defaults13 to13 a13 search13 of13 all13 records13 ()13 in13 the13 last13 1513 minutes13 and13 returns13 the13 results13
13
More13 complex13 searches13 are13 possible13 by13 changing13 the13 me13 filter13 and13 supplying13 search13 queries13
Change13 the13 0me13 filter13 To13 change13 the13 me13 ranges13 click13 the13 clock13 symbol13 in13 the13 top-shy‐right13 of13 the13 Kibana13 Discover13 page13
13
You13 will13 see13 the13 following13 opons13
13
Page of 22 71 Page of 22 71
A13 variety13 of13 quick13 me13 ranges13 can13 be13 selected13 or13 you13 can13 switch13 to13 relave13 or13 absolute13 me13 ranges13
Quick13 me13 ranges13 allow13 switching13 between13 last13 X13 minutes13 to13 last13 Y13 hours13 or13 Z13 days13 etc13
Relave13 me13 ranges13 allow13 more13 specific13 versions13 of13 the13 Quick13 opons13 from13 specific13 minuteshoursdays13 to13 now13
13
Absolute13 me13 ranges13 specify13 the13 start13 and13 end13 mes13 for13 the13 searches13
13
Custom13 search13 queries13 You13 can13 use13 the13 text13 window13 of13 the13 Discover13 page13 to13 search13 for13 matches13 within13 the13 log13 records13 You13 can13 use13 Apache13 Lucene13 or13 Elascsearch13 query13 DSL13 syntax13
13
Page of 23 71 Page of 23 71
Saved13 searches13 You13 can13 save13 any13 custom13 search13 syntax13 and13 choice13 of13 me13 range13 for13 future13 ease13 of13 use13 Any13 relave13 or13 absolute13 me13 frame13 is13 saved13 as13 part13 of13 the13 search13 13
To13 save13 a13 search13
1 Set13 up13 and13 run13 the13 required13 syntax13 and13 me13 frame13
13
2 Click13 the13 Save13 Search13 icon13
13
3 Type13 in13 a13 name13 for13 the13 saved13 search13
Page of 24 71 Page of 24 71
13
7 To13 delete13 or13 update13 saved13 searches13 click13 13 manage13 saved13 searches13
13
Use13 the13 edit13 icons13 to13 make13 any13 changes13
Page of 26 71 Page of 26 71
13
Build13 simple13 Kibana13 histograms13 Kibana13 allows13 you13 to13 build13 charts13 and13 dashboards13 based13 on13 the13 search13 capability13 offered13 by13 the13 Discover13 page13 Saved13 searches13 can13 be13 used13 as13 the13 basis13 for13 charts13 and13 dashboards13 built13 using13 those13 charts13 To13 build13 a13 simple13 histogram13
1 Select13 the13 Visualize13 page13
13
2 Select13 Ver0cal13 Bar13 Chart13
13
3 To13 use13 a13 saved13 search13 click13 From13 a13 saved13 search13
Page of 27 71 Page of 27 71
13
4 Select13 a13 previously13 saved13 search13
13
5 Select13 to13 edit13 the13 x-shy‐axis
6 Edit13 the13 fields13 with13 informaon13 about13 the13 aggregaon13 field13 interval13 and13 custom13 label13
Page of 28 71 Page of 28 71
13
Page of 29 71 Page of 29 71
7 Click13 the13 green13 arrow13 The13 histogram13 is13 created13 and13 run13 based13 on13 the13 saved13 search
13
8 To13 save13 the13 histogram13 click13 the13 Save13 Visualiza0on13 icon13 and13 add13 a13 tle13
13
13
9 Click13 Save13
10 To13 view13 the13 visualisaon13 click13 the13 Dashboard13 page13
Page of 30 71 Page of 30 71
13
11 Click13 the13 Add13 Visualiza0on13 icon13
13
12 Select13 the13 visiualizaon13 that13 you13 just13 created13
13
Page of 31 71 Page of 31 71
13
13 To13 save13 the13 dashboard13 click13 the13 Save13 Dashboard13 icon13
Page of 32 71 Page of 32 71
13
14 Add13 a13 tle13 and13 select13 whether13 to13 store13 the13 selected13 me13 frame13 with13 the13 dashboard13
13
15 Click13 Save13
16 You13 can13 edit13 the13 shape13 and13 number13 of13 charts13 and13 save13 the13 dashboard13
17 To13 re-shy‐run13 the13 dashboard13 select13 the13 Dashboard13 page13 and13 click13 the13 Load13 Saved13 Dashboard13 icon13
Page of 33 71 Page of 33 71
13
18 Select13 the13 saved13 dashboard13 from13 the13 list13
13
19 The13 dashboard13 runs13
13
Page of 34 71 Page of 34 71
Page of 35 71 Page of 35 71
Aler0ng13 with13 ELK13 ELK13 examines13 the13 log13 records13 in13 process13 to13 send13 alerts13 to13 third-shy‐party13 alerng13 event13 management13 and13 social13 media13 applicaons13 As13 shown13 previously13 in13 the13 ELK13 architecture13 the13 ELK13 strategy13 for13 alerng13 is13 to13 send13 alerts13 from13 the13 Logstash13 component13 directly13 Therefore13 log13 records13 and13 alerts13 are13 sent13 to13 both13 Elascsearch13 and13 third-shy‐party13 tools13 at13 the13 same13 instant13 using13 the13 output13 secon13 of13 the13 Logstash13 configuraon13 Any13 alerts13 are13 generated13 during13 the13 filter13 secon13 processing13 of13 the13 Logstash13 configuraon13
Create13 alerts13 in13 Logstash13 The13 Logstash13 filter13 secon13 is13 used13 to13 trigger13 alerts13 based13 on13 the13 log13 records13 being13 processed13 For13 example13 log13 records13 can13 be13 checked13 for13 errors13 warnings13 crashed13 or13 stopped13 messages13 and13 generate13 an13 alert13
Here13 is13 an13 example13 Logstash13 filter13 code13
13 if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
The13 specific13 tags13 are13 set13 against13 the13 log13 records13 to13 trigger13 the13 alert13 later13 in13 the13 configuraon13 in13 the13 output13 secon13
Control13 the13 number13 of13 generated13 alerts13 using13 the13 Logstash13 throcle13 plugin13 so13 that13 either13 a13 lot13 of13 instances13 in13 a13 fixed13 me13 period13 are13 required13 before13 an13 alert13 is13 sent13 or13 a13 specific13 number13 of13 instances13 triggers13 the13 alert13
Page of 36 71 Page of 36 71
Edit13 the13 plugin13 parameters13 to13 meet13 system13 requirements13
13 13 13 13 13 if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 10
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle13
after_count =gt 20
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 30
period =gt 1200
key =gt servicename_alarm_warning
Page of 37 71 Page of 37 71
add_tag =gt servicename_throttled
13 13 13 13 13 13 13 13 13
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 40
period =gt 1500
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
Enrich13 alerts13 with13 message13 detail13 Some13 of13 the13 third-shy‐party13 integraons13 benefit13 from13 having13 components13 of13 the13 raw13 log13 record13 sent13 in13 addion13 to13 the13 raw13 record13 or13 fields13 in13 the13 record13 edited13 transformed13 or13 translated13
Grok13 Logstash13 allows13 the13 log13 record13 to13 be13 broken13 up13 into13 logical13 secons13 and13 each13 secon13 stored13 in13 a13 new13 parameter13 using13 the13 grok13 parameter13 in13 the13 filter13 secon13 Here13 is13 an13 example13 grok13
13 grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
To13 build13 the13 grok13 statement13 Logstash13 offers13 an13 online13 tool13 at13 hcpsgrokdebugherokuappcom13
Translate13 If13 specific13 fields13 in13 the13 log13 record13 are13 coded13 (for13 example13 the13 Bluemix13 app13 name13 is13 oaen13 not13 sent13 in13 the13 syslog13 messages13 instead13 the13 GUID13 is13 sent13 which13 is13 not13 useful)13 then13 the13 filter13 secon13 translate13 plugin13 can13 be13 used13 to13 add13 new13 fields13 by13 holding13 the13 decoded13 versions13 of13 those13 values13
13 translate
field =gt data1
destination =gt data1
override =gt true
Page of 38 71 Page of 38 71
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
dc50f6e4-c052-436a-bf27-95ba392c4228 monverifyservicemanagement
0c677e82-5bfc-4e38-9155-e416249e3ee6 ms-catalog-postpyretic-pomiculture
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
However13 the13 translate13 plugin13 is13 not13 included13 with13 the13 Logstash13 installaon13 In13 the13 next13 secon13 I13 will13 13 describe13 how13 to13 add13 new13 plugins13 to13 the13 Logstash13 instance13
Add13 custom13 plugins13 to13 Logstash13 As13 an13 open13 source13 tool13 Logstash13 encourages13 users13 to13 develop13 content13 for13 all13 users13 to13 share13 Some13 of13 the13 plugins13 that13 were13 created13 by13 our13 community13 include13 a13 translaon13 plugin13 an13 output13 Slack13 plugin13 and13 an13 output13 syslog13 plugin13 Letrsquos13 look13 at13 how13 to13 add13 these13 plugins13 to13 your13 environment13
Filter13 translate13 plugin13 To13 add13 the13 translate13 plugin13
1 Download13 the13 plugin13
2 Copy13 the13 plugin13 file13 logstash-shy‐filter-shy‐translate-shy‐214gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-filter-translate 214
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Output13 Slack13 plugin13 To13 add13 the13 Slack13 plugin13
Page of 39 71 Page of 39 71
1 Download13 the13 plugin13 13
2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐slack-shy‐014gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-output-slack 014
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Output13 syslog13 plugin13 To13 add13 the13 syslog13 plugin13
1 Download13 the13 plugin13 13
2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐syslog-shy‐214gem13 to13 the13 ELK13 Virtual13 Server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-output-syslog 214
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Integrate13 Logstash13 with13 Slack13 To13 send13 generated13 alerts13 to13 Slack13 complete13 these13 steps13
1 Create13 a13 Slack13 integraon13 13
2 Use13 the13 Slack13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 Slack13 integraon13
113 Login13 to13 Slack13 and13 go13 to13 the13 Apps13 page13 13
Page of 40 71 Page of 40 71
13
313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13
13
A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13
13
Page of 42 71 Page of 42 71
The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13
13
The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13
13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
slack
url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5
channel =gt cloud-operations-mon
username =gt nick_cawood
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13
Page of 43 71 Page of 43 71
Page of 44 71 Page of 44 71
In13 Slack13 the13 alerts13 look13 like13 this13
13
Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13
1 Create13 a13 PagerDuty13 integraon13 13
2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13
1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13
13
2 Click13 Add13 New13 Service13
13
313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13
Page of 45 71 Page of 45 71
13
413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13
513 Click13 Add13 Service13
13
The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13
Page of 46 71 Page of 46 71
13
The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt be7eb1ea9c224018923e818d3e891d0b
incident_key =gt logstashhosttype
Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13
Page of 47 71 Page of 47 71
13
Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13
1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13
2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13
3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13
4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13
13 13 http
url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo
http_method =gt post
format =gt json
For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13
13 13 syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
Page of 48 71 Page of 48 71
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13
13
The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13
if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2
This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13
Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13
Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13
Logstash13 severity Shown13 in13 Omnibus13 as
alert Indeterminate
crical Warning
debug 7
emergency Clear
error Minor
informaonal 6
noce Crical
warning Major
Page of 49 71 Page of 49 71
13
Page of 50 71 Page of 50 71
Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13
input
tcp
port =gt 5000
type =gt syslog
filter
mutate
add_field =gt format =gt cf
grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
translate
field =gt data1
destination =gt data1
override =gt true
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
=== Check Alarm Criteria ===
Standard Key Words
if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
=== Alarm Exclusions ===
if [message] =~ (i)sometexthere
mutate remove_tag =gt servicename_alarm
Throttle alarms so we do not hit PD rate limits
Page of 52 71 Page of 52 71
if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 5
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle
after_count =gt 5
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 5
period =gt 1200
key =gt servicename_alarm_warning
add_tag =gt servicename_throttled
Page of 53 71 Page of 53 71
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
output
elasticsearch
hosts =gt localhost
stdout codec =gt rubydebug
if servicename_alarm_err in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
Page of 54 71 Page of 54 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 55 71 Page of 55 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
Page of 56 71 Page of 56 71
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert err message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_error in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
Page of 57 71 Page of 57 71
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 58 71 Page of 58 71
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 59 71 Page of 59 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert error message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
Page of 60 71 Page of 60 71
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
Page of 61 71 Page of 61 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 62 71 Page of 62 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt x
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert fatal message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
Page of 63 71 Page of 63 71
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
Page of 64 71 Page of 64 71
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 65 71 Page of 65 71
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt heavy_exclamation_mark
Page of 66 71 Page of 66 71
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert WARNING message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 67 71 Page of 67 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
Page of 68 71 Page of 68 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 69 71 Page of 69 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
Page of 70 71 Page of 70 71
Page of 71 71 Page of 71 71
[rootelkforcase]$ cd config
[rootelkforcase]$ touch loggregatorconf
Edit13 the13 loggregator13 file13 with13 the13 following13 contents13
13 Input13 secon13 defines13 which13 log13 records13 to13 receive13
13 port13 and13 type13 reflect13 method13 selected13 for13 Cloud13 Foundry13 to13 broadcast13 the13 records13
input
tcp
port =gt 5000
type =gt syslog
13 Filter13 secon13 ndash13 transforms13 the13 data13
filter
Add a field to identify the records as Cloud Foundry based
mutate
add_field =gt format =gt cf
13 Output13 secon13 ndash13 sends13 the13 data13 to13 its13 desnaon(s)13
output
13 Use13 the13 elascsearch13 plugin13 to13 send13 the13 data13 to13 the13 elascsearch13 instance13
13 elasticsearch
hosts =gt localhost
stdout codec =gt rubydebug
13 End13 of13 Config13
Save13 the13 file13 Verify13 the13 format13 of13 the13 file13 is13 acceptable13 by13 running13 the13 following13 commands13
[rootelkforcase]$ cd homelogstashlogstash-233
[rootelkforcase]$ binlogstash agent --configtest -f configloggregatorconf
Page of 20 71 Page of 20 71
Configuraon13 is13 OK13
Start13 Logstash13 Aaer13 you13 have13 wricen13 and13 verified13 the13 Logstash13 configuraon13 you13 can13 start13 Logstash13 to13 process13 records13 from13 the13 Bluemix13 apps13 (via13 Cloud13 Foundry)13 Run13 the13 following13 as13 root13 user13
[rootelkforcase]$ cd homelogstashlogstash-233
[rootelkforcase]$ binlogstash agent -f configloggregatorconf
Logstash13 is13 now13 started13
Monitor13 Logstash13 processing13 Once13 Logstash13 is13 running13 and13 log13 records13 are13 being13 produced13 by13 the13 Bluemix13 apps13 you13 can13 monitor13 progress13 in13 the13 same13 window13 that13 was13 used13 to13 start13 Logstash13 Output13 is13 similar13 to13 the13 following13
13
The13 log13 records13 being13 broadcast13 from13 the13 Bluemix13 apps13 are13 now13 sent13 to13 Elascsearch13 from13 the13 Logstash13 output13 configuraon13 Records13 are13 sent13 in13 real13 me13 If13 Logstash13 is13 not13 running13 while13 log13 records13 are13 being13 produced13 then13 these13 records13 will13 not13 be13 processed13 by13 Logstash13 and13 will13 not13 be13 stored13 in13 Elascsearch13
Page of 21 71 Page of 21 71
Use13 ELK13 The13 log13 records13 from13 the13 Bluemix13 apps13 are13 now13 being13 stored13 and13 indexed13 by13 the13 Elascsearch13 component13 This13 data13 can13 be13 accessed13 via13 the13 Kibana13 applicaon13
Kibana13 Discover13 page13 Using13 a13 web13 browser13 either13 directly13 from13 a13 client13 or13 using13 the13 VNC13 client13 access13 Kibana13 using13 the13 publicIPaddress13 of13 the13 ELK13 virtual13 server13
hcpltpublicIPaddressgt560113
No13 userpassword13 credenals13 are13 required13
Kibana13 defaults13 to13 a13 search13 of13 all13 records13 ()13 in13 the13 last13 1513 minutes13 and13 returns13 the13 results13
13
More13 complex13 searches13 are13 possible13 by13 changing13 the13 me13 filter13 and13 supplying13 search13 queries13
Change13 the13 0me13 filter13 To13 change13 the13 me13 ranges13 click13 the13 clock13 symbol13 in13 the13 top-shy‐right13 of13 the13 Kibana13 Discover13 page13
13
You13 will13 see13 the13 following13 opons13
13
Page of 22 71 Page of 22 71
A13 variety13 of13 quick13 me13 ranges13 can13 be13 selected13 or13 you13 can13 switch13 to13 relave13 or13 absolute13 me13 ranges13
Quick13 me13 ranges13 allow13 switching13 between13 last13 X13 minutes13 to13 last13 Y13 hours13 or13 Z13 days13 etc13
Relave13 me13 ranges13 allow13 more13 specific13 versions13 of13 the13 Quick13 opons13 from13 specific13 minuteshoursdays13 to13 now13
13
Absolute13 me13 ranges13 specify13 the13 start13 and13 end13 mes13 for13 the13 searches13
13
Custom13 search13 queries13 You13 can13 use13 the13 text13 window13 of13 the13 Discover13 page13 to13 search13 for13 matches13 within13 the13 log13 records13 You13 can13 use13 Apache13 Lucene13 or13 Elascsearch13 query13 DSL13 syntax13
13
Page of 23 71 Page of 23 71
Saved13 searches13 You13 can13 save13 any13 custom13 search13 syntax13 and13 choice13 of13 me13 range13 for13 future13 ease13 of13 use13 Any13 relave13 or13 absolute13 me13 frame13 is13 saved13 as13 part13 of13 the13 search13 13
To13 save13 a13 search13
1 Set13 up13 and13 run13 the13 required13 syntax13 and13 me13 frame13
13
2 Click13 the13 Save13 Search13 icon13
13
3 Type13 in13 a13 name13 for13 the13 saved13 search13
Page of 24 71 Page of 24 71
13
7 To13 delete13 or13 update13 saved13 searches13 click13 13 manage13 saved13 searches13
13
Use13 the13 edit13 icons13 to13 make13 any13 changes13
Page of 26 71 Page of 26 71
13
Build13 simple13 Kibana13 histograms13 Kibana13 allows13 you13 to13 build13 charts13 and13 dashboards13 based13 on13 the13 search13 capability13 offered13 by13 the13 Discover13 page13 Saved13 searches13 can13 be13 used13 as13 the13 basis13 for13 charts13 and13 dashboards13 built13 using13 those13 charts13 To13 build13 a13 simple13 histogram13
1 Select13 the13 Visualize13 page13
13
2 Select13 Ver0cal13 Bar13 Chart13
13
3 To13 use13 a13 saved13 search13 click13 From13 a13 saved13 search13
Page of 27 71 Page of 27 71
13
4 Select13 a13 previously13 saved13 search13
13
5 Select13 to13 edit13 the13 x-shy‐axis
6 Edit13 the13 fields13 with13 informaon13 about13 the13 aggregaon13 field13 interval13 and13 custom13 label13
Page of 28 71 Page of 28 71
13
Page of 29 71 Page of 29 71
7 Click13 the13 green13 arrow13 The13 histogram13 is13 created13 and13 run13 based13 on13 the13 saved13 search
13
8 To13 save13 the13 histogram13 click13 the13 Save13 Visualiza0on13 icon13 and13 add13 a13 tle13
13
13
9 Click13 Save13
10 To13 view13 the13 visualisaon13 click13 the13 Dashboard13 page13
Page of 30 71 Page of 30 71
13
11 Click13 the13 Add13 Visualiza0on13 icon13
13
12 Select13 the13 visiualizaon13 that13 you13 just13 created13
13
Page of 31 71 Page of 31 71
13
13 To13 save13 the13 dashboard13 click13 the13 Save13 Dashboard13 icon13
Page of 32 71 Page of 32 71
13
14 Add13 a13 tle13 and13 select13 whether13 to13 store13 the13 selected13 me13 frame13 with13 the13 dashboard13
13
15 Click13 Save13
16 You13 can13 edit13 the13 shape13 and13 number13 of13 charts13 and13 save13 the13 dashboard13
17 To13 re-shy‐run13 the13 dashboard13 select13 the13 Dashboard13 page13 and13 click13 the13 Load13 Saved13 Dashboard13 icon13
Page of 33 71 Page of 33 71
13
18 Select13 the13 saved13 dashboard13 from13 the13 list13
13
19 The13 dashboard13 runs13
13
Page of 34 71 Page of 34 71
Page of 35 71 Page of 35 71
Aler0ng13 with13 ELK13 ELK13 examines13 the13 log13 records13 in13 process13 to13 send13 alerts13 to13 third-shy‐party13 alerng13 event13 management13 and13 social13 media13 applicaons13 As13 shown13 previously13 in13 the13 ELK13 architecture13 the13 ELK13 strategy13 for13 alerng13 is13 to13 send13 alerts13 from13 the13 Logstash13 component13 directly13 Therefore13 log13 records13 and13 alerts13 are13 sent13 to13 both13 Elascsearch13 and13 third-shy‐party13 tools13 at13 the13 same13 instant13 using13 the13 output13 secon13 of13 the13 Logstash13 configuraon13 Any13 alerts13 are13 generated13 during13 the13 filter13 secon13 processing13 of13 the13 Logstash13 configuraon13
Create13 alerts13 in13 Logstash13 The13 Logstash13 filter13 secon13 is13 used13 to13 trigger13 alerts13 based13 on13 the13 log13 records13 being13 processed13 For13 example13 log13 records13 can13 be13 checked13 for13 errors13 warnings13 crashed13 or13 stopped13 messages13 and13 generate13 an13 alert13
Here13 is13 an13 example13 Logstash13 filter13 code13
13 if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
The13 specific13 tags13 are13 set13 against13 the13 log13 records13 to13 trigger13 the13 alert13 later13 in13 the13 configuraon13 in13 the13 output13 secon13
Control13 the13 number13 of13 generated13 alerts13 using13 the13 Logstash13 throcle13 plugin13 so13 that13 either13 a13 lot13 of13 instances13 in13 a13 fixed13 me13 period13 are13 required13 before13 an13 alert13 is13 sent13 or13 a13 specific13 number13 of13 instances13 triggers13 the13 alert13
Page of 36 71 Page of 36 71
Edit13 the13 plugin13 parameters13 to13 meet13 system13 requirements13
13 13 13 13 13 if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 10
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle13
after_count =gt 20
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 30
period =gt 1200
key =gt servicename_alarm_warning
Page of 37 71 Page of 37 71
add_tag =gt servicename_throttled
13 13 13 13 13 13 13 13 13
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 40
period =gt 1500
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
Enrich13 alerts13 with13 message13 detail13 Some13 of13 the13 third-shy‐party13 integraons13 benefit13 from13 having13 components13 of13 the13 raw13 log13 record13 sent13 in13 addion13 to13 the13 raw13 record13 or13 fields13 in13 the13 record13 edited13 transformed13 or13 translated13
Grok13 Logstash13 allows13 the13 log13 record13 to13 be13 broken13 up13 into13 logical13 secons13 and13 each13 secon13 stored13 in13 a13 new13 parameter13 using13 the13 grok13 parameter13 in13 the13 filter13 secon13 Here13 is13 an13 example13 grok13
13 grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
To13 build13 the13 grok13 statement13 Logstash13 offers13 an13 online13 tool13 at13 hcpsgrokdebugherokuappcom13
Translate13 If13 specific13 fields13 in13 the13 log13 record13 are13 coded13 (for13 example13 the13 Bluemix13 app13 name13 is13 oaen13 not13 sent13 in13 the13 syslog13 messages13 instead13 the13 GUID13 is13 sent13 which13 is13 not13 useful)13 then13 the13 filter13 secon13 translate13 plugin13 can13 be13 used13 to13 add13 new13 fields13 by13 holding13 the13 decoded13 versions13 of13 those13 values13
13 translate
field =gt data1
destination =gt data1
override =gt true
Page of 38 71 Page of 38 71
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
dc50f6e4-c052-436a-bf27-95ba392c4228 monverifyservicemanagement
0c677e82-5bfc-4e38-9155-e416249e3ee6 ms-catalog-postpyretic-pomiculture
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
However13 the13 translate13 plugin13 is13 not13 included13 with13 the13 Logstash13 installaon13 In13 the13 next13 secon13 I13 will13 13 describe13 how13 to13 add13 new13 plugins13 to13 the13 Logstash13 instance13
Add13 custom13 plugins13 to13 Logstash13 As13 an13 open13 source13 tool13 Logstash13 encourages13 users13 to13 develop13 content13 for13 all13 users13 to13 share13 Some13 of13 the13 plugins13 that13 were13 created13 by13 our13 community13 include13 a13 translaon13 plugin13 an13 output13 Slack13 plugin13 and13 an13 output13 syslog13 plugin13 Letrsquos13 look13 at13 how13 to13 add13 these13 plugins13 to13 your13 environment13
Filter13 translate13 plugin13 To13 add13 the13 translate13 plugin13
1 Download13 the13 plugin13
2 Copy13 the13 plugin13 file13 logstash-shy‐filter-shy‐translate-shy‐214gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-filter-translate 214
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Output13 Slack13 plugin13 To13 add13 the13 Slack13 plugin13
Page of 39 71 Page of 39 71
1 Download13 the13 plugin13 13
2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐slack-shy‐014gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-output-slack 014
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Output13 syslog13 plugin13 To13 add13 the13 syslog13 plugin13
1 Download13 the13 plugin13 13
2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐syslog-shy‐214gem13 to13 the13 ELK13 Virtual13 Server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-output-syslog 214
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Integrate13 Logstash13 with13 Slack13 To13 send13 generated13 alerts13 to13 Slack13 complete13 these13 steps13
1 Create13 a13 Slack13 integraon13 13
2 Use13 the13 Slack13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 Slack13 integraon13
113 Login13 to13 Slack13 and13 go13 to13 the13 Apps13 page13 13
Page of 40 71 Page of 40 71
13
313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13
13
A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13
13
Page of 42 71 Page of 42 71
The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13
13
The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13
13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
slack
url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5
channel =gt cloud-operations-mon
username =gt nick_cawood
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13
Page of 43 71 Page of 43 71
Page of 44 71 Page of 44 71
In13 Slack13 the13 alerts13 look13 like13 this13
13
Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13
1 Create13 a13 PagerDuty13 integraon13 13
2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13
1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13
13
2 Click13 Add13 New13 Service13
13
313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13
Page of 45 71 Page of 45 71
13
413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13
513 Click13 Add13 Service13
13
The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13
Page of 46 71 Page of 46 71
13
The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt be7eb1ea9c224018923e818d3e891d0b
incident_key =gt logstashhosttype
Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13
Page of 47 71 Page of 47 71
13
Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13
1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13
2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13
3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13
4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13
13 13 http
url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo
http_method =gt post
format =gt json
For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13
13 13 syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
Page of 48 71 Page of 48 71
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13
13
The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13
if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2
This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13
Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13
Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13
Logstash13 severity Shown13 in13 Omnibus13 as
alert Indeterminate
crical Warning
debug 7
emergency Clear
error Minor
informaonal 6
noce Crical
warning Major
Page of 49 71 Page of 49 71
13
Page of 50 71 Page of 50 71
Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13
input
tcp
port =gt 5000
type =gt syslog
filter
mutate
add_field =gt format =gt cf
grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
translate
field =gt data1
destination =gt data1
override =gt true
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
=== Check Alarm Criteria ===
Standard Key Words
if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
=== Alarm Exclusions ===
if [message] =~ (i)sometexthere
mutate remove_tag =gt servicename_alarm
Throttle alarms so we do not hit PD rate limits
Page of 52 71 Page of 52 71
if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 5
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle
after_count =gt 5
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 5
period =gt 1200
key =gt servicename_alarm_warning
add_tag =gt servicename_throttled
Page of 53 71 Page of 53 71
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
output
elasticsearch
hosts =gt localhost
stdout codec =gt rubydebug
if servicename_alarm_err in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
Page of 54 71 Page of 54 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 55 71 Page of 55 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
Page of 56 71 Page of 56 71
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert err message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_error in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
Page of 57 71 Page of 57 71
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 58 71 Page of 58 71
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 59 71 Page of 59 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert error message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
Page of 60 71 Page of 60 71
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
Page of 61 71 Page of 61 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 62 71 Page of 62 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt x
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert fatal message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
Page of 63 71 Page of 63 71
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
Page of 64 71 Page of 64 71
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 65 71 Page of 65 71
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt heavy_exclamation_mark
Page of 66 71 Page of 66 71
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert WARNING message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 67 71 Page of 67 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
Page of 68 71 Page of 68 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 69 71 Page of 69 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
Page of 70 71 Page of 70 71
Page of 71 71 Page of 71 71
Configuraon13 is13 OK13
Start13 Logstash13 Aaer13 you13 have13 wricen13 and13 verified13 the13 Logstash13 configuraon13 you13 can13 start13 Logstash13 to13 process13 records13 from13 the13 Bluemix13 apps13 (via13 Cloud13 Foundry)13 Run13 the13 following13 as13 root13 user13
[rootelkforcase]$ cd homelogstashlogstash-233
[rootelkforcase]$ binlogstash agent -f configloggregatorconf
Logstash13 is13 now13 started13
Monitor13 Logstash13 processing13 Once13 Logstash13 is13 running13 and13 log13 records13 are13 being13 produced13 by13 the13 Bluemix13 apps13 you13 can13 monitor13 progress13 in13 the13 same13 window13 that13 was13 used13 to13 start13 Logstash13 Output13 is13 similar13 to13 the13 following13
13
The13 log13 records13 being13 broadcast13 from13 the13 Bluemix13 apps13 are13 now13 sent13 to13 Elascsearch13 from13 the13 Logstash13 output13 configuraon13 Records13 are13 sent13 in13 real13 me13 If13 Logstash13 is13 not13 running13 while13 log13 records13 are13 being13 produced13 then13 these13 records13 will13 not13 be13 processed13 by13 Logstash13 and13 will13 not13 be13 stored13 in13 Elascsearch13
Page of 21 71 Page of 21 71
Use13 ELK13 The13 log13 records13 from13 the13 Bluemix13 apps13 are13 now13 being13 stored13 and13 indexed13 by13 the13 Elascsearch13 component13 This13 data13 can13 be13 accessed13 via13 the13 Kibana13 applicaon13
Kibana13 Discover13 page13 Using13 a13 web13 browser13 either13 directly13 from13 a13 client13 or13 using13 the13 VNC13 client13 access13 Kibana13 using13 the13 publicIPaddress13 of13 the13 ELK13 virtual13 server13
hcpltpublicIPaddressgt560113
No13 userpassword13 credenals13 are13 required13
Kibana13 defaults13 to13 a13 search13 of13 all13 records13 ()13 in13 the13 last13 1513 minutes13 and13 returns13 the13 results13
13
More13 complex13 searches13 are13 possible13 by13 changing13 the13 me13 filter13 and13 supplying13 search13 queries13
Change13 the13 0me13 filter13 To13 change13 the13 me13 ranges13 click13 the13 clock13 symbol13 in13 the13 top-shy‐right13 of13 the13 Kibana13 Discover13 page13
13
You13 will13 see13 the13 following13 opons13
13
Page of 22 71 Page of 22 71
A13 variety13 of13 quick13 me13 ranges13 can13 be13 selected13 or13 you13 can13 switch13 to13 relave13 or13 absolute13 me13 ranges13
Quick13 me13 ranges13 allow13 switching13 between13 last13 X13 minutes13 to13 last13 Y13 hours13 or13 Z13 days13 etc13
Relave13 me13 ranges13 allow13 more13 specific13 versions13 of13 the13 Quick13 opons13 from13 specific13 minuteshoursdays13 to13 now13
13
Absolute13 me13 ranges13 specify13 the13 start13 and13 end13 mes13 for13 the13 searches13
13
Custom13 search13 queries13 You13 can13 use13 the13 text13 window13 of13 the13 Discover13 page13 to13 search13 for13 matches13 within13 the13 log13 records13 You13 can13 use13 Apache13 Lucene13 or13 Elascsearch13 query13 DSL13 syntax13
13
Page of 23 71 Page of 23 71
Saved13 searches13 You13 can13 save13 any13 custom13 search13 syntax13 and13 choice13 of13 me13 range13 for13 future13 ease13 of13 use13 Any13 relave13 or13 absolute13 me13 frame13 is13 saved13 as13 part13 of13 the13 search13 13
To13 save13 a13 search13
1 Set13 up13 and13 run13 the13 required13 syntax13 and13 me13 frame13
13
2 Click13 the13 Save13 Search13 icon13
13
3 Type13 in13 a13 name13 for13 the13 saved13 search13
Page of 24 71 Page of 24 71
13
7 To13 delete13 or13 update13 saved13 searches13 click13 13 manage13 saved13 searches13
13
Use13 the13 edit13 icons13 to13 make13 any13 changes13
Page of 26 71 Page of 26 71
13
Build13 simple13 Kibana13 histograms13 Kibana13 allows13 you13 to13 build13 charts13 and13 dashboards13 based13 on13 the13 search13 capability13 offered13 by13 the13 Discover13 page13 Saved13 searches13 can13 be13 used13 as13 the13 basis13 for13 charts13 and13 dashboards13 built13 using13 those13 charts13 To13 build13 a13 simple13 histogram13
1 Select13 the13 Visualize13 page13
13
2 Select13 Ver0cal13 Bar13 Chart13
13
3 To13 use13 a13 saved13 search13 click13 From13 a13 saved13 search13
Page of 27 71 Page of 27 71
13
4 Select13 a13 previously13 saved13 search13
13
5 Select13 to13 edit13 the13 x-shy‐axis
6 Edit13 the13 fields13 with13 informaon13 about13 the13 aggregaon13 field13 interval13 and13 custom13 label13
Page of 28 71 Page of 28 71
13
Page of 29 71 Page of 29 71
7 Click13 the13 green13 arrow13 The13 histogram13 is13 created13 and13 run13 based13 on13 the13 saved13 search
13
8 To13 save13 the13 histogram13 click13 the13 Save13 Visualiza0on13 icon13 and13 add13 a13 tle13
13
13
9 Click13 Save13
10 To13 view13 the13 visualisaon13 click13 the13 Dashboard13 page13
Page of 30 71 Page of 30 71
13
11 Click13 the13 Add13 Visualiza0on13 icon13
13
12 Select13 the13 visiualizaon13 that13 you13 just13 created13
13
Page of 31 71 Page of 31 71
13
13 To13 save13 the13 dashboard13 click13 the13 Save13 Dashboard13 icon13
Page of 32 71 Page of 32 71
13
14 Add13 a13 tle13 and13 select13 whether13 to13 store13 the13 selected13 me13 frame13 with13 the13 dashboard13
13
15 Click13 Save13
16 You13 can13 edit13 the13 shape13 and13 number13 of13 charts13 and13 save13 the13 dashboard13
17 To13 re-shy‐run13 the13 dashboard13 select13 the13 Dashboard13 page13 and13 click13 the13 Load13 Saved13 Dashboard13 icon13
Page of 33 71 Page of 33 71
13
18 Select13 the13 saved13 dashboard13 from13 the13 list13
13
19 The13 dashboard13 runs13
13
Page of 34 71 Page of 34 71
Page of 35 71 Page of 35 71
Aler0ng13 with13 ELK13 ELK13 examines13 the13 log13 records13 in13 process13 to13 send13 alerts13 to13 third-shy‐party13 alerng13 event13 management13 and13 social13 media13 applicaons13 As13 shown13 previously13 in13 the13 ELK13 architecture13 the13 ELK13 strategy13 for13 alerng13 is13 to13 send13 alerts13 from13 the13 Logstash13 component13 directly13 Therefore13 log13 records13 and13 alerts13 are13 sent13 to13 both13 Elascsearch13 and13 third-shy‐party13 tools13 at13 the13 same13 instant13 using13 the13 output13 secon13 of13 the13 Logstash13 configuraon13 Any13 alerts13 are13 generated13 during13 the13 filter13 secon13 processing13 of13 the13 Logstash13 configuraon13
Create13 alerts13 in13 Logstash13 The13 Logstash13 filter13 secon13 is13 used13 to13 trigger13 alerts13 based13 on13 the13 log13 records13 being13 processed13 For13 example13 log13 records13 can13 be13 checked13 for13 errors13 warnings13 crashed13 or13 stopped13 messages13 and13 generate13 an13 alert13
Here13 is13 an13 example13 Logstash13 filter13 code13
13 if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
The13 specific13 tags13 are13 set13 against13 the13 log13 records13 to13 trigger13 the13 alert13 later13 in13 the13 configuraon13 in13 the13 output13 secon13
Control13 the13 number13 of13 generated13 alerts13 using13 the13 Logstash13 throcle13 plugin13 so13 that13 either13 a13 lot13 of13 instances13 in13 a13 fixed13 me13 period13 are13 required13 before13 an13 alert13 is13 sent13 or13 a13 specific13 number13 of13 instances13 triggers13 the13 alert13
Page of 36 71 Page of 36 71
Edit13 the13 plugin13 parameters13 to13 meet13 system13 requirements13
13 13 13 13 13 if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 10
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle13
after_count =gt 20
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 30
period =gt 1200
key =gt servicename_alarm_warning
Page of 37 71 Page of 37 71
add_tag =gt servicename_throttled
13 13 13 13 13 13 13 13 13
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 40
period =gt 1500
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
Enrich13 alerts13 with13 message13 detail13 Some13 of13 the13 third-shy‐party13 integraons13 benefit13 from13 having13 components13 of13 the13 raw13 log13 record13 sent13 in13 addion13 to13 the13 raw13 record13 or13 fields13 in13 the13 record13 edited13 transformed13 or13 translated13
Grok13 Logstash13 allows13 the13 log13 record13 to13 be13 broken13 up13 into13 logical13 secons13 and13 each13 secon13 stored13 in13 a13 new13 parameter13 using13 the13 grok13 parameter13 in13 the13 filter13 secon13 Here13 is13 an13 example13 grok13
13 grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
To13 build13 the13 grok13 statement13 Logstash13 offers13 an13 online13 tool13 at13 hcpsgrokdebugherokuappcom13
Translate13 If13 specific13 fields13 in13 the13 log13 record13 are13 coded13 (for13 example13 the13 Bluemix13 app13 name13 is13 oaen13 not13 sent13 in13 the13 syslog13 messages13 instead13 the13 GUID13 is13 sent13 which13 is13 not13 useful)13 then13 the13 filter13 secon13 translate13 plugin13 can13 be13 used13 to13 add13 new13 fields13 by13 holding13 the13 decoded13 versions13 of13 those13 values13
13 translate
field =gt data1
destination =gt data1
override =gt true
Page of 38 71 Page of 38 71
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
dc50f6e4-c052-436a-bf27-95ba392c4228 monverifyservicemanagement
0c677e82-5bfc-4e38-9155-e416249e3ee6 ms-catalog-postpyretic-pomiculture
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
However13 the13 translate13 plugin13 is13 not13 included13 with13 the13 Logstash13 installaon13 In13 the13 next13 secon13 I13 will13 13 describe13 how13 to13 add13 new13 plugins13 to13 the13 Logstash13 instance13
Add13 custom13 plugins13 to13 Logstash13 As13 an13 open13 source13 tool13 Logstash13 encourages13 users13 to13 develop13 content13 for13 all13 users13 to13 share13 Some13 of13 the13 plugins13 that13 were13 created13 by13 our13 community13 include13 a13 translaon13 plugin13 an13 output13 Slack13 plugin13 and13 an13 output13 syslog13 plugin13 Letrsquos13 look13 at13 how13 to13 add13 these13 plugins13 to13 your13 environment13
Filter13 translate13 plugin13 To13 add13 the13 translate13 plugin13
1 Download13 the13 plugin13
2 Copy13 the13 plugin13 file13 logstash-shy‐filter-shy‐translate-shy‐214gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-filter-translate 214
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Output13 Slack13 plugin13 To13 add13 the13 Slack13 plugin13
Page of 39 71 Page of 39 71
1 Download13 the13 plugin13 13
2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐slack-shy‐014gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-output-slack 014
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Output13 syslog13 plugin13 To13 add13 the13 syslog13 plugin13
1 Download13 the13 plugin13 13
2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐syslog-shy‐214gem13 to13 the13 ELK13 Virtual13 Server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-output-syslog 214
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Integrate13 Logstash13 with13 Slack13 To13 send13 generated13 alerts13 to13 Slack13 complete13 these13 steps13
1 Create13 a13 Slack13 integraon13 13
2 Use13 the13 Slack13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 Slack13 integraon13
113 Login13 to13 Slack13 and13 go13 to13 the13 Apps13 page13 13
Page of 40 71 Page of 40 71
13
313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13
13
A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13
13
Page of 42 71 Page of 42 71
The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13
13
The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13
13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
slack
url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5
channel =gt cloud-operations-mon
username =gt nick_cawood
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13
Page of 43 71 Page of 43 71
Page of 44 71 Page of 44 71
In13 Slack13 the13 alerts13 look13 like13 this13
13
Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13
1 Create13 a13 PagerDuty13 integraon13 13
2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13
1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13
13
2 Click13 Add13 New13 Service13
13
313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13
Page of 45 71 Page of 45 71
13
413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13
513 Click13 Add13 Service13
13
The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13
Page of 46 71 Page of 46 71
13
The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt be7eb1ea9c224018923e818d3e891d0b
incident_key =gt logstashhosttype
Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13
Page of 47 71 Page of 47 71
13
Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13
1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13
2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13
3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13
4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13
13 13 http
url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo
http_method =gt post
format =gt json
For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13
13 13 syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
Page of 48 71 Page of 48 71
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13
13
The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13
if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2
This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13
Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13
Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13
Logstash13 severity Shown13 in13 Omnibus13 as
alert Indeterminate
crical Warning
debug 7
emergency Clear
error Minor
informaonal 6
noce Crical
warning Major
Page of 49 71 Page of 49 71
13
Page of 50 71 Page of 50 71
Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13
input
tcp
port =gt 5000
type =gt syslog
filter
mutate
add_field =gt format =gt cf
grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
translate
field =gt data1
destination =gt data1
override =gt true
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
=== Check Alarm Criteria ===
Standard Key Words
if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
=== Alarm Exclusions ===
if [message] =~ (i)sometexthere
mutate remove_tag =gt servicename_alarm
Throttle alarms so we do not hit PD rate limits
Page of 52 71 Page of 52 71
if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 5
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle
after_count =gt 5
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 5
period =gt 1200
key =gt servicename_alarm_warning
add_tag =gt servicename_throttled
Page of 53 71 Page of 53 71
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
output
elasticsearch
hosts =gt localhost
stdout codec =gt rubydebug
if servicename_alarm_err in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
Page of 54 71 Page of 54 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 55 71 Page of 55 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
Page of 56 71 Page of 56 71
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert err message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_error in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
Page of 57 71 Page of 57 71
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 58 71 Page of 58 71
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 59 71 Page of 59 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert error message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
Page of 60 71 Page of 60 71
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
Page of 61 71 Page of 61 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 62 71 Page of 62 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt x
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert fatal message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
Page of 63 71 Page of 63 71
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
Page of 64 71 Page of 64 71
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 65 71 Page of 65 71
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt heavy_exclamation_mark
Page of 66 71 Page of 66 71
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert WARNING message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 67 71 Page of 67 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
Page of 68 71 Page of 68 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 69 71 Page of 69 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
Page of 70 71 Page of 70 71
Page of 71 71 Page of 71 71
Use13 ELK13 The13 log13 records13 from13 the13 Bluemix13 apps13 are13 now13 being13 stored13 and13 indexed13 by13 the13 Elascsearch13 component13 This13 data13 can13 be13 accessed13 via13 the13 Kibana13 applicaon13
Kibana13 Discover13 page13 Using13 a13 web13 browser13 either13 directly13 from13 a13 client13 or13 using13 the13 VNC13 client13 access13 Kibana13 using13 the13 publicIPaddress13 of13 the13 ELK13 virtual13 server13
hcpltpublicIPaddressgt560113
No13 userpassword13 credenals13 are13 required13
Kibana13 defaults13 to13 a13 search13 of13 all13 records13 ()13 in13 the13 last13 1513 minutes13 and13 returns13 the13 results13
13
More13 complex13 searches13 are13 possible13 by13 changing13 the13 me13 filter13 and13 supplying13 search13 queries13
Change13 the13 0me13 filter13 To13 change13 the13 me13 ranges13 click13 the13 clock13 symbol13 in13 the13 top-shy‐right13 of13 the13 Kibana13 Discover13 page13
13
You13 will13 see13 the13 following13 opons13
13
Page of 22 71 Page of 22 71
A13 variety13 of13 quick13 me13 ranges13 can13 be13 selected13 or13 you13 can13 switch13 to13 relave13 or13 absolute13 me13 ranges13
Quick13 me13 ranges13 allow13 switching13 between13 last13 X13 minutes13 to13 last13 Y13 hours13 or13 Z13 days13 etc13
Relave13 me13 ranges13 allow13 more13 specific13 versions13 of13 the13 Quick13 opons13 from13 specific13 minuteshoursdays13 to13 now13
13
Absolute13 me13 ranges13 specify13 the13 start13 and13 end13 mes13 for13 the13 searches13
13
Custom13 search13 queries13 You13 can13 use13 the13 text13 window13 of13 the13 Discover13 page13 to13 search13 for13 matches13 within13 the13 log13 records13 You13 can13 use13 Apache13 Lucene13 or13 Elascsearch13 query13 DSL13 syntax13
13
Page of 23 71 Page of 23 71
Saved13 searches13 You13 can13 save13 any13 custom13 search13 syntax13 and13 choice13 of13 me13 range13 for13 future13 ease13 of13 use13 Any13 relave13 or13 absolute13 me13 frame13 is13 saved13 as13 part13 of13 the13 search13 13
To13 save13 a13 search13
1 Set13 up13 and13 run13 the13 required13 syntax13 and13 me13 frame13
13
2 Click13 the13 Save13 Search13 icon13
13
3 Type13 in13 a13 name13 for13 the13 saved13 search13
Page of 24 71 Page of 24 71
13
7 To13 delete13 or13 update13 saved13 searches13 click13 13 manage13 saved13 searches13
13
Use13 the13 edit13 icons13 to13 make13 any13 changes13
Page of 26 71 Page of 26 71
13
Build13 simple13 Kibana13 histograms13 Kibana13 allows13 you13 to13 build13 charts13 and13 dashboards13 based13 on13 the13 search13 capability13 offered13 by13 the13 Discover13 page13 Saved13 searches13 can13 be13 used13 as13 the13 basis13 for13 charts13 and13 dashboards13 built13 using13 those13 charts13 To13 build13 a13 simple13 histogram13
1 Select13 the13 Visualize13 page13
13
2 Select13 Ver0cal13 Bar13 Chart13
13
3 To13 use13 a13 saved13 search13 click13 From13 a13 saved13 search13
Page of 27 71 Page of 27 71
13
4 Select13 a13 previously13 saved13 search13
13
5 Select13 to13 edit13 the13 x-shy‐axis
6 Edit13 the13 fields13 with13 informaon13 about13 the13 aggregaon13 field13 interval13 and13 custom13 label13
Page of 28 71 Page of 28 71
13
Page of 29 71 Page of 29 71
7 Click13 the13 green13 arrow13 The13 histogram13 is13 created13 and13 run13 based13 on13 the13 saved13 search
13
8 To13 save13 the13 histogram13 click13 the13 Save13 Visualiza0on13 icon13 and13 add13 a13 tle13
13
13
9 Click13 Save13
10 To13 view13 the13 visualisaon13 click13 the13 Dashboard13 page13
Page of 30 71 Page of 30 71
13
11 Click13 the13 Add13 Visualiza0on13 icon13
13
12 Select13 the13 visiualizaon13 that13 you13 just13 created13
13
Page of 31 71 Page of 31 71
13
13 To13 save13 the13 dashboard13 click13 the13 Save13 Dashboard13 icon13
Page of 32 71 Page of 32 71
13
14 Add13 a13 tle13 and13 select13 whether13 to13 store13 the13 selected13 me13 frame13 with13 the13 dashboard13
13
15 Click13 Save13
16 You13 can13 edit13 the13 shape13 and13 number13 of13 charts13 and13 save13 the13 dashboard13
17 To13 re-shy‐run13 the13 dashboard13 select13 the13 Dashboard13 page13 and13 click13 the13 Load13 Saved13 Dashboard13 icon13
Page of 33 71 Page of 33 71
13
18 Select13 the13 saved13 dashboard13 from13 the13 list13
13
19 The13 dashboard13 runs13
13
Page of 34 71 Page of 34 71
Page of 35 71 Page of 35 71
Aler0ng13 with13 ELK13 ELK13 examines13 the13 log13 records13 in13 process13 to13 send13 alerts13 to13 third-shy‐party13 alerng13 event13 management13 and13 social13 media13 applicaons13 As13 shown13 previously13 in13 the13 ELK13 architecture13 the13 ELK13 strategy13 for13 alerng13 is13 to13 send13 alerts13 from13 the13 Logstash13 component13 directly13 Therefore13 log13 records13 and13 alerts13 are13 sent13 to13 both13 Elascsearch13 and13 third-shy‐party13 tools13 at13 the13 same13 instant13 using13 the13 output13 secon13 of13 the13 Logstash13 configuraon13 Any13 alerts13 are13 generated13 during13 the13 filter13 secon13 processing13 of13 the13 Logstash13 configuraon13
Create13 alerts13 in13 Logstash13 The13 Logstash13 filter13 secon13 is13 used13 to13 trigger13 alerts13 based13 on13 the13 log13 records13 being13 processed13 For13 example13 log13 records13 can13 be13 checked13 for13 errors13 warnings13 crashed13 or13 stopped13 messages13 and13 generate13 an13 alert13
Here13 is13 an13 example13 Logstash13 filter13 code13
13 if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
The13 specific13 tags13 are13 set13 against13 the13 log13 records13 to13 trigger13 the13 alert13 later13 in13 the13 configuraon13 in13 the13 output13 secon13
Control13 the13 number13 of13 generated13 alerts13 using13 the13 Logstash13 throcle13 plugin13 so13 that13 either13 a13 lot13 of13 instances13 in13 a13 fixed13 me13 period13 are13 required13 before13 an13 alert13 is13 sent13 or13 a13 specific13 number13 of13 instances13 triggers13 the13 alert13
Page of 36 71 Page of 36 71
Edit13 the13 plugin13 parameters13 to13 meet13 system13 requirements13
13 13 13 13 13 if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 10
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle13
after_count =gt 20
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 30
period =gt 1200
key =gt servicename_alarm_warning
Page of 37 71 Page of 37 71
add_tag =gt servicename_throttled
13 13 13 13 13 13 13 13 13
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 40
period =gt 1500
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
Enrich13 alerts13 with13 message13 detail13 Some13 of13 the13 third-shy‐party13 integraons13 benefit13 from13 having13 components13 of13 the13 raw13 log13 record13 sent13 in13 addion13 to13 the13 raw13 record13 or13 fields13 in13 the13 record13 edited13 transformed13 or13 translated13
Grok13 Logstash13 allows13 the13 log13 record13 to13 be13 broken13 up13 into13 logical13 secons13 and13 each13 secon13 stored13 in13 a13 new13 parameter13 using13 the13 grok13 parameter13 in13 the13 filter13 secon13 Here13 is13 an13 example13 grok13
13 grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
To13 build13 the13 grok13 statement13 Logstash13 offers13 an13 online13 tool13 at13 hcpsgrokdebugherokuappcom13
Translate13 If13 specific13 fields13 in13 the13 log13 record13 are13 coded13 (for13 example13 the13 Bluemix13 app13 name13 is13 oaen13 not13 sent13 in13 the13 syslog13 messages13 instead13 the13 GUID13 is13 sent13 which13 is13 not13 useful)13 then13 the13 filter13 secon13 translate13 plugin13 can13 be13 used13 to13 add13 new13 fields13 by13 holding13 the13 decoded13 versions13 of13 those13 values13
13 translate
field =gt data1
destination =gt data1
override =gt true
Page of 38 71 Page of 38 71
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
dc50f6e4-c052-436a-bf27-95ba392c4228 monverifyservicemanagement
0c677e82-5bfc-4e38-9155-e416249e3ee6 ms-catalog-postpyretic-pomiculture
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
However13 the13 translate13 plugin13 is13 not13 included13 with13 the13 Logstash13 installaon13 In13 the13 next13 secon13 I13 will13 13 describe13 how13 to13 add13 new13 plugins13 to13 the13 Logstash13 instance13
Add13 custom13 plugins13 to13 Logstash13 As13 an13 open13 source13 tool13 Logstash13 encourages13 users13 to13 develop13 content13 for13 all13 users13 to13 share13 Some13 of13 the13 plugins13 that13 were13 created13 by13 our13 community13 include13 a13 translaon13 plugin13 an13 output13 Slack13 plugin13 and13 an13 output13 syslog13 plugin13 Letrsquos13 look13 at13 how13 to13 add13 these13 plugins13 to13 your13 environment13
Filter13 translate13 plugin13 To13 add13 the13 translate13 plugin13
1 Download13 the13 plugin13
2 Copy13 the13 plugin13 file13 logstash-shy‐filter-shy‐translate-shy‐214gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-filter-translate 214
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Output13 Slack13 plugin13 To13 add13 the13 Slack13 plugin13
Page of 39 71 Page of 39 71
1 Download13 the13 plugin13 13
2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐slack-shy‐014gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-output-slack 014
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Output13 syslog13 plugin13 To13 add13 the13 syslog13 plugin13
1 Download13 the13 plugin13 13
2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐syslog-shy‐214gem13 to13 the13 ELK13 Virtual13 Server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-output-syslog 214
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Integrate13 Logstash13 with13 Slack13 To13 send13 generated13 alerts13 to13 Slack13 complete13 these13 steps13
1 Create13 a13 Slack13 integraon13 13
2 Use13 the13 Slack13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 Slack13 integraon13
113 Login13 to13 Slack13 and13 go13 to13 the13 Apps13 page13 13
Page of 40 71 Page of 40 71
13
313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13
13
A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13
13
Page of 42 71 Page of 42 71
The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13
13
The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13
13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
slack
url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5
channel =gt cloud-operations-mon
username =gt nick_cawood
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13
Page of 43 71 Page of 43 71
Page of 44 71 Page of 44 71
In13 Slack13 the13 alerts13 look13 like13 this13
13
Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13
1 Create13 a13 PagerDuty13 integraon13 13
2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13
1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13
13
2 Click13 Add13 New13 Service13
13
313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13
Page of 45 71 Page of 45 71
13
413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13
513 Click13 Add13 Service13
13
The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13
Page of 46 71 Page of 46 71
13
The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt be7eb1ea9c224018923e818d3e891d0b
incident_key =gt logstashhosttype
Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13
Page of 47 71 Page of 47 71
13
Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13
1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13
2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13
3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13
4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13
13 13 http
url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo
http_method =gt post
format =gt json
For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13
13 13 syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
Page of 48 71 Page of 48 71
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13
13
The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13
if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2
This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13
Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13
Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13
Logstash13 severity Shown13 in13 Omnibus13 as
alert Indeterminate
crical Warning
debug 7
emergency Clear
error Minor
informaonal 6
noce Crical
warning Major
Page of 49 71 Page of 49 71
13
Page of 50 71 Page of 50 71
Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13
input
tcp
port =gt 5000
type =gt syslog
filter
mutate
add_field =gt format =gt cf
grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
translate
field =gt data1
destination =gt data1
override =gt true
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
=== Check Alarm Criteria ===
Standard Key Words
if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
=== Alarm Exclusions ===
if [message] =~ (i)sometexthere
mutate remove_tag =gt servicename_alarm
Throttle alarms so we do not hit PD rate limits
Page of 52 71 Page of 52 71
if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 5
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle
after_count =gt 5
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 5
period =gt 1200
key =gt servicename_alarm_warning
add_tag =gt servicename_throttled
Page of 53 71 Page of 53 71
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
output
elasticsearch
hosts =gt localhost
stdout codec =gt rubydebug
if servicename_alarm_err in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
Page of 54 71 Page of 54 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 55 71 Page of 55 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
Page of 56 71 Page of 56 71
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert err message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_error in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
Page of 57 71 Page of 57 71
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 58 71 Page of 58 71
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 59 71 Page of 59 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert error message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
Page of 60 71 Page of 60 71
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
Page of 61 71 Page of 61 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 62 71 Page of 62 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt x
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert fatal message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
Page of 63 71 Page of 63 71
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
Page of 64 71 Page of 64 71
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 65 71 Page of 65 71
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt heavy_exclamation_mark
Page of 66 71 Page of 66 71
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert WARNING message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 67 71 Page of 67 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
Page of 68 71 Page of 68 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 69 71 Page of 69 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
Page of 70 71 Page of 70 71
Page of 71 71 Page of 71 71
A13 variety13 of13 quick13 me13 ranges13 can13 be13 selected13 or13 you13 can13 switch13 to13 relave13 or13 absolute13 me13 ranges13
Quick13 me13 ranges13 allow13 switching13 between13 last13 X13 minutes13 to13 last13 Y13 hours13 or13 Z13 days13 etc13
Relave13 me13 ranges13 allow13 more13 specific13 versions13 of13 the13 Quick13 opons13 from13 specific13 minuteshoursdays13 to13 now13
13
Absolute13 me13 ranges13 specify13 the13 start13 and13 end13 mes13 for13 the13 searches13
13
Custom13 search13 queries13 You13 can13 use13 the13 text13 window13 of13 the13 Discover13 page13 to13 search13 for13 matches13 within13 the13 log13 records13 You13 can13 use13 Apache13 Lucene13 or13 Elascsearch13 query13 DSL13 syntax13
13
Page of 23 71 Page of 23 71
Saved13 searches13 You13 can13 save13 any13 custom13 search13 syntax13 and13 choice13 of13 me13 range13 for13 future13 ease13 of13 use13 Any13 relave13 or13 absolute13 me13 frame13 is13 saved13 as13 part13 of13 the13 search13 13
To13 save13 a13 search13
1 Set13 up13 and13 run13 the13 required13 syntax13 and13 me13 frame13
13
2 Click13 the13 Save13 Search13 icon13
13
3 Type13 in13 a13 name13 for13 the13 saved13 search13
Page of 24 71 Page of 24 71
13
7 To13 delete13 or13 update13 saved13 searches13 click13 13 manage13 saved13 searches13
13
Use13 the13 edit13 icons13 to13 make13 any13 changes13
Page of 26 71 Page of 26 71
13
Build13 simple13 Kibana13 histograms13 Kibana13 allows13 you13 to13 build13 charts13 and13 dashboards13 based13 on13 the13 search13 capability13 offered13 by13 the13 Discover13 page13 Saved13 searches13 can13 be13 used13 as13 the13 basis13 for13 charts13 and13 dashboards13 built13 using13 those13 charts13 To13 build13 a13 simple13 histogram13
1 Select13 the13 Visualize13 page13
13
2 Select13 Ver0cal13 Bar13 Chart13
13
3 To13 use13 a13 saved13 search13 click13 From13 a13 saved13 search13
Page of 27 71 Page of 27 71
13
4 Select13 a13 previously13 saved13 search13
13
5 Select13 to13 edit13 the13 x-shy‐axis
6 Edit13 the13 fields13 with13 informaon13 about13 the13 aggregaon13 field13 interval13 and13 custom13 label13
Page of 28 71 Page of 28 71
13
Page of 29 71 Page of 29 71
7 Click13 the13 green13 arrow13 The13 histogram13 is13 created13 and13 run13 based13 on13 the13 saved13 search
13
8 To13 save13 the13 histogram13 click13 the13 Save13 Visualiza0on13 icon13 and13 add13 a13 tle13
13
13
9 Click13 Save13
10 To13 view13 the13 visualisaon13 click13 the13 Dashboard13 page13
Page of 30 71 Page of 30 71
13
11 Click13 the13 Add13 Visualiza0on13 icon13
13
12 Select13 the13 visiualizaon13 that13 you13 just13 created13
13
Page of 31 71 Page of 31 71
13
13 To13 save13 the13 dashboard13 click13 the13 Save13 Dashboard13 icon13
Page of 32 71 Page of 32 71
13
14 Add13 a13 tle13 and13 select13 whether13 to13 store13 the13 selected13 me13 frame13 with13 the13 dashboard13
13
15 Click13 Save13
16 You13 can13 edit13 the13 shape13 and13 number13 of13 charts13 and13 save13 the13 dashboard13
17 To13 re-shy‐run13 the13 dashboard13 select13 the13 Dashboard13 page13 and13 click13 the13 Load13 Saved13 Dashboard13 icon13
Page of 33 71 Page of 33 71
13
18 Select13 the13 saved13 dashboard13 from13 the13 list13
13
19 The13 dashboard13 runs13
13
Page of 34 71 Page of 34 71
Page of 35 71 Page of 35 71
Aler0ng13 with13 ELK13 ELK13 examines13 the13 log13 records13 in13 process13 to13 send13 alerts13 to13 third-shy‐party13 alerng13 event13 management13 and13 social13 media13 applicaons13 As13 shown13 previously13 in13 the13 ELK13 architecture13 the13 ELK13 strategy13 for13 alerng13 is13 to13 send13 alerts13 from13 the13 Logstash13 component13 directly13 Therefore13 log13 records13 and13 alerts13 are13 sent13 to13 both13 Elascsearch13 and13 third-shy‐party13 tools13 at13 the13 same13 instant13 using13 the13 output13 secon13 of13 the13 Logstash13 configuraon13 Any13 alerts13 are13 generated13 during13 the13 filter13 secon13 processing13 of13 the13 Logstash13 configuraon13
Create13 alerts13 in13 Logstash13 The13 Logstash13 filter13 secon13 is13 used13 to13 trigger13 alerts13 based13 on13 the13 log13 records13 being13 processed13 For13 example13 log13 records13 can13 be13 checked13 for13 errors13 warnings13 crashed13 or13 stopped13 messages13 and13 generate13 an13 alert13
Here13 is13 an13 example13 Logstash13 filter13 code13
13 if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
The13 specific13 tags13 are13 set13 against13 the13 log13 records13 to13 trigger13 the13 alert13 later13 in13 the13 configuraon13 in13 the13 output13 secon13
Control13 the13 number13 of13 generated13 alerts13 using13 the13 Logstash13 throcle13 plugin13 so13 that13 either13 a13 lot13 of13 instances13 in13 a13 fixed13 me13 period13 are13 required13 before13 an13 alert13 is13 sent13 or13 a13 specific13 number13 of13 instances13 triggers13 the13 alert13
Page of 36 71 Page of 36 71
Edit13 the13 plugin13 parameters13 to13 meet13 system13 requirements13
13 13 13 13 13 if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 10
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle13
after_count =gt 20
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 30
period =gt 1200
key =gt servicename_alarm_warning
Page of 37 71 Page of 37 71
add_tag =gt servicename_throttled
13 13 13 13 13 13 13 13 13
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 40
period =gt 1500
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
Enrich13 alerts13 with13 message13 detail13 Some13 of13 the13 third-shy‐party13 integraons13 benefit13 from13 having13 components13 of13 the13 raw13 log13 record13 sent13 in13 addion13 to13 the13 raw13 record13 or13 fields13 in13 the13 record13 edited13 transformed13 or13 translated13
Grok13 Logstash13 allows13 the13 log13 record13 to13 be13 broken13 up13 into13 logical13 secons13 and13 each13 secon13 stored13 in13 a13 new13 parameter13 using13 the13 grok13 parameter13 in13 the13 filter13 secon13 Here13 is13 an13 example13 grok13
13 grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
To13 build13 the13 grok13 statement13 Logstash13 offers13 an13 online13 tool13 at13 hcpsgrokdebugherokuappcom13
Translate13 If13 specific13 fields13 in13 the13 log13 record13 are13 coded13 (for13 example13 the13 Bluemix13 app13 name13 is13 oaen13 not13 sent13 in13 the13 syslog13 messages13 instead13 the13 GUID13 is13 sent13 which13 is13 not13 useful)13 then13 the13 filter13 secon13 translate13 plugin13 can13 be13 used13 to13 add13 new13 fields13 by13 holding13 the13 decoded13 versions13 of13 those13 values13
13 translate
field =gt data1
destination =gt data1
override =gt true
Page of 38 71 Page of 38 71
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
dc50f6e4-c052-436a-bf27-95ba392c4228 monverifyservicemanagement
0c677e82-5bfc-4e38-9155-e416249e3ee6 ms-catalog-postpyretic-pomiculture
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
However13 the13 translate13 plugin13 is13 not13 included13 with13 the13 Logstash13 installaon13 In13 the13 next13 secon13 I13 will13 13 describe13 how13 to13 add13 new13 plugins13 to13 the13 Logstash13 instance13
Add13 custom13 plugins13 to13 Logstash13 As13 an13 open13 source13 tool13 Logstash13 encourages13 users13 to13 develop13 content13 for13 all13 users13 to13 share13 Some13 of13 the13 plugins13 that13 were13 created13 by13 our13 community13 include13 a13 translaon13 plugin13 an13 output13 Slack13 plugin13 and13 an13 output13 syslog13 plugin13 Letrsquos13 look13 at13 how13 to13 add13 these13 plugins13 to13 your13 environment13
Filter13 translate13 plugin13 To13 add13 the13 translate13 plugin13
1 Download13 the13 plugin13
2 Copy13 the13 plugin13 file13 logstash-shy‐filter-shy‐translate-shy‐214gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-filter-translate 214
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Output13 Slack13 plugin13 To13 add13 the13 Slack13 plugin13
Page of 39 71 Page of 39 71
1 Download13 the13 plugin13 13
2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐slack-shy‐014gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-output-slack 014
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Output13 syslog13 plugin13 To13 add13 the13 syslog13 plugin13
1 Download13 the13 plugin13 13
2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐syslog-shy‐214gem13 to13 the13 ELK13 Virtual13 Server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-output-syslog 214
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Integrate13 Logstash13 with13 Slack13 To13 send13 generated13 alerts13 to13 Slack13 complete13 these13 steps13
1 Create13 a13 Slack13 integraon13 13
2 Use13 the13 Slack13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 Slack13 integraon13
113 Login13 to13 Slack13 and13 go13 to13 the13 Apps13 page13 13
Page of 40 71 Page of 40 71
13
313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13
13
A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13
13
Page of 42 71 Page of 42 71
The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13
13
The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13
13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
slack
url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5
channel =gt cloud-operations-mon
username =gt nick_cawood
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13
Page of 43 71 Page of 43 71
Page of 44 71 Page of 44 71
In13 Slack13 the13 alerts13 look13 like13 this13
13
Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13
1 Create13 a13 PagerDuty13 integraon13 13
2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13
1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13
13
2 Click13 Add13 New13 Service13
13
313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13
Page of 45 71 Page of 45 71
13
413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13
513 Click13 Add13 Service13
13
The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13
Page of 46 71 Page of 46 71
13
The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt be7eb1ea9c224018923e818d3e891d0b
incident_key =gt logstashhosttype
Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13
Page of 47 71 Page of 47 71
13
Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13
1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13
2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13
3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13
4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13
13 13 http
url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo
http_method =gt post
format =gt json
For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13
13 13 syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
Page of 48 71 Page of 48 71
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13
13
The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13
if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2
This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13
Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13
Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13
Logstash13 severity Shown13 in13 Omnibus13 as
alert Indeterminate
crical Warning
debug 7
emergency Clear
error Minor
informaonal 6
noce Crical
warning Major
Page of 49 71 Page of 49 71
13
Page of 50 71 Page of 50 71
Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13
input
tcp
port =gt 5000
type =gt syslog
filter
mutate
add_field =gt format =gt cf
grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
translate
field =gt data1
destination =gt data1
override =gt true
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
=== Check Alarm Criteria ===
Standard Key Words
if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
=== Alarm Exclusions ===
if [message] =~ (i)sometexthere
mutate remove_tag =gt servicename_alarm
Throttle alarms so we do not hit PD rate limits
Page of 52 71 Page of 52 71
if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 5
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle
after_count =gt 5
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 5
period =gt 1200
key =gt servicename_alarm_warning
add_tag =gt servicename_throttled
Page of 53 71 Page of 53 71
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
output
elasticsearch
hosts =gt localhost
stdout codec =gt rubydebug
if servicename_alarm_err in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
Page of 54 71 Page of 54 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 55 71 Page of 55 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
Page of 56 71 Page of 56 71
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert err message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_error in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
Page of 57 71 Page of 57 71
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 58 71 Page of 58 71
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 59 71 Page of 59 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert error message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
Page of 60 71 Page of 60 71
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
Page of 61 71 Page of 61 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 62 71 Page of 62 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt x
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert fatal message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
Page of 63 71 Page of 63 71
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
Page of 64 71 Page of 64 71
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 65 71 Page of 65 71
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt heavy_exclamation_mark
Page of 66 71 Page of 66 71
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert WARNING message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 67 71 Page of 67 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
Page of 68 71 Page of 68 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 69 71 Page of 69 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
Page of 70 71 Page of 70 71
Page of 71 71 Page of 71 71
Saved13 searches13 You13 can13 save13 any13 custom13 search13 syntax13 and13 choice13 of13 me13 range13 for13 future13 ease13 of13 use13 Any13 relave13 or13 absolute13 me13 frame13 is13 saved13 as13 part13 of13 the13 search13 13
To13 save13 a13 search13
1 Set13 up13 and13 run13 the13 required13 syntax13 and13 me13 frame13
13
2 Click13 the13 Save13 Search13 icon13
13
3 Type13 in13 a13 name13 for13 the13 saved13 search13
Page of 24 71 Page of 24 71
13
7 To13 delete13 or13 update13 saved13 searches13 click13 13 manage13 saved13 searches13
13
Use13 the13 edit13 icons13 to13 make13 any13 changes13
Page of 26 71 Page of 26 71
13
Build13 simple13 Kibana13 histograms13 Kibana13 allows13 you13 to13 build13 charts13 and13 dashboards13 based13 on13 the13 search13 capability13 offered13 by13 the13 Discover13 page13 Saved13 searches13 can13 be13 used13 as13 the13 basis13 for13 charts13 and13 dashboards13 built13 using13 those13 charts13 To13 build13 a13 simple13 histogram13
1 Select13 the13 Visualize13 page13
13
2 Select13 Ver0cal13 Bar13 Chart13
13
3 To13 use13 a13 saved13 search13 click13 From13 a13 saved13 search13
Page of 27 71 Page of 27 71
13
4 Select13 a13 previously13 saved13 search13
13
5 Select13 to13 edit13 the13 x-shy‐axis
6 Edit13 the13 fields13 with13 informaon13 about13 the13 aggregaon13 field13 interval13 and13 custom13 label13
Page of 28 71 Page of 28 71
13
Page of 29 71 Page of 29 71
7 Click13 the13 green13 arrow13 The13 histogram13 is13 created13 and13 run13 based13 on13 the13 saved13 search
13
8 To13 save13 the13 histogram13 click13 the13 Save13 Visualiza0on13 icon13 and13 add13 a13 tle13
13
13
9 Click13 Save13
10 To13 view13 the13 visualisaon13 click13 the13 Dashboard13 page13
Page of 30 71 Page of 30 71
13
11 Click13 the13 Add13 Visualiza0on13 icon13
13
12 Select13 the13 visiualizaon13 that13 you13 just13 created13
13
Page of 31 71 Page of 31 71
13
13 To13 save13 the13 dashboard13 click13 the13 Save13 Dashboard13 icon13
Page of 32 71 Page of 32 71
13
14 Add13 a13 tle13 and13 select13 whether13 to13 store13 the13 selected13 me13 frame13 with13 the13 dashboard13
13
15 Click13 Save13
16 You13 can13 edit13 the13 shape13 and13 number13 of13 charts13 and13 save13 the13 dashboard13
17 To13 re-shy‐run13 the13 dashboard13 select13 the13 Dashboard13 page13 and13 click13 the13 Load13 Saved13 Dashboard13 icon13
Page of 33 71 Page of 33 71
13
18 Select13 the13 saved13 dashboard13 from13 the13 list13
13
19 The13 dashboard13 runs13
13
Page of 34 71 Page of 34 71
Page of 35 71 Page of 35 71
Aler0ng13 with13 ELK13 ELK13 examines13 the13 log13 records13 in13 process13 to13 send13 alerts13 to13 third-shy‐party13 alerng13 event13 management13 and13 social13 media13 applicaons13 As13 shown13 previously13 in13 the13 ELK13 architecture13 the13 ELK13 strategy13 for13 alerng13 is13 to13 send13 alerts13 from13 the13 Logstash13 component13 directly13 Therefore13 log13 records13 and13 alerts13 are13 sent13 to13 both13 Elascsearch13 and13 third-shy‐party13 tools13 at13 the13 same13 instant13 using13 the13 output13 secon13 of13 the13 Logstash13 configuraon13 Any13 alerts13 are13 generated13 during13 the13 filter13 secon13 processing13 of13 the13 Logstash13 configuraon13
Create13 alerts13 in13 Logstash13 The13 Logstash13 filter13 secon13 is13 used13 to13 trigger13 alerts13 based13 on13 the13 log13 records13 being13 processed13 For13 example13 log13 records13 can13 be13 checked13 for13 errors13 warnings13 crashed13 or13 stopped13 messages13 and13 generate13 an13 alert13
Here13 is13 an13 example13 Logstash13 filter13 code13
13 if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
The13 specific13 tags13 are13 set13 against13 the13 log13 records13 to13 trigger13 the13 alert13 later13 in13 the13 configuraon13 in13 the13 output13 secon13
Control13 the13 number13 of13 generated13 alerts13 using13 the13 Logstash13 throcle13 plugin13 so13 that13 either13 a13 lot13 of13 instances13 in13 a13 fixed13 me13 period13 are13 required13 before13 an13 alert13 is13 sent13 or13 a13 specific13 number13 of13 instances13 triggers13 the13 alert13
Page of 36 71 Page of 36 71
Edit13 the13 plugin13 parameters13 to13 meet13 system13 requirements13
13 13 13 13 13 if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 10
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle13
after_count =gt 20
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 30
period =gt 1200
key =gt servicename_alarm_warning
Page of 37 71 Page of 37 71
add_tag =gt servicename_throttled
13 13 13 13 13 13 13 13 13
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 40
period =gt 1500
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
Enrich13 alerts13 with13 message13 detail13 Some13 of13 the13 third-shy‐party13 integraons13 benefit13 from13 having13 components13 of13 the13 raw13 log13 record13 sent13 in13 addion13 to13 the13 raw13 record13 or13 fields13 in13 the13 record13 edited13 transformed13 or13 translated13
Grok13 Logstash13 allows13 the13 log13 record13 to13 be13 broken13 up13 into13 logical13 secons13 and13 each13 secon13 stored13 in13 a13 new13 parameter13 using13 the13 grok13 parameter13 in13 the13 filter13 secon13 Here13 is13 an13 example13 grok13
13 grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
To13 build13 the13 grok13 statement13 Logstash13 offers13 an13 online13 tool13 at13 hcpsgrokdebugherokuappcom13
Translate13 If13 specific13 fields13 in13 the13 log13 record13 are13 coded13 (for13 example13 the13 Bluemix13 app13 name13 is13 oaen13 not13 sent13 in13 the13 syslog13 messages13 instead13 the13 GUID13 is13 sent13 which13 is13 not13 useful)13 then13 the13 filter13 secon13 translate13 plugin13 can13 be13 used13 to13 add13 new13 fields13 by13 holding13 the13 decoded13 versions13 of13 those13 values13
13 translate
field =gt data1
destination =gt data1
override =gt true
Page of 38 71 Page of 38 71
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
dc50f6e4-c052-436a-bf27-95ba392c4228 monverifyservicemanagement
0c677e82-5bfc-4e38-9155-e416249e3ee6 ms-catalog-postpyretic-pomiculture
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
However13 the13 translate13 plugin13 is13 not13 included13 with13 the13 Logstash13 installaon13 In13 the13 next13 secon13 I13 will13 13 describe13 how13 to13 add13 new13 plugins13 to13 the13 Logstash13 instance13
Add13 custom13 plugins13 to13 Logstash13 As13 an13 open13 source13 tool13 Logstash13 encourages13 users13 to13 develop13 content13 for13 all13 users13 to13 share13 Some13 of13 the13 plugins13 that13 were13 created13 by13 our13 community13 include13 a13 translaon13 plugin13 an13 output13 Slack13 plugin13 and13 an13 output13 syslog13 plugin13 Letrsquos13 look13 at13 how13 to13 add13 these13 plugins13 to13 your13 environment13
Filter13 translate13 plugin13 To13 add13 the13 translate13 plugin13
1 Download13 the13 plugin13
2 Copy13 the13 plugin13 file13 logstash-shy‐filter-shy‐translate-shy‐214gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-filter-translate 214
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Output13 Slack13 plugin13 To13 add13 the13 Slack13 plugin13
Page of 39 71 Page of 39 71
1 Download13 the13 plugin13 13
2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐slack-shy‐014gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-output-slack 014
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Output13 syslog13 plugin13 To13 add13 the13 syslog13 plugin13
1 Download13 the13 plugin13 13
2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐syslog-shy‐214gem13 to13 the13 ELK13 Virtual13 Server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-output-syslog 214
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Integrate13 Logstash13 with13 Slack13 To13 send13 generated13 alerts13 to13 Slack13 complete13 these13 steps13
1 Create13 a13 Slack13 integraon13 13
2 Use13 the13 Slack13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 Slack13 integraon13
113 Login13 to13 Slack13 and13 go13 to13 the13 Apps13 page13 13
Page of 40 71 Page of 40 71
13
313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13
13
A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13
13
Page of 42 71 Page of 42 71
The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13
13
The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13
13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
slack
url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5
channel =gt cloud-operations-mon
username =gt nick_cawood
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13
Page of 43 71 Page of 43 71
Page of 44 71 Page of 44 71
In13 Slack13 the13 alerts13 look13 like13 this13
13
Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13
1 Create13 a13 PagerDuty13 integraon13 13
2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13
1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13
13
2 Click13 Add13 New13 Service13
13
313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13
Page of 45 71 Page of 45 71
13
413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13
513 Click13 Add13 Service13
13
The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13
Page of 46 71 Page of 46 71
13
The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt be7eb1ea9c224018923e818d3e891d0b
incident_key =gt logstashhosttype
Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13
Page of 47 71 Page of 47 71
13
Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13
1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13
2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13
3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13
4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13
13 13 http
url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo
http_method =gt post
format =gt json
For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13
13 13 syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
Page of 48 71 Page of 48 71
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13
13
The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13
if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2
This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13
Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13
Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13
Logstash13 severity Shown13 in13 Omnibus13 as
alert Indeterminate
crical Warning
debug 7
emergency Clear
error Minor
informaonal 6
noce Crical
warning Major
Page of 49 71 Page of 49 71
13
Page of 50 71 Page of 50 71
Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13
input
tcp
port =gt 5000
type =gt syslog
filter
mutate
add_field =gt format =gt cf
grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
translate
field =gt data1
destination =gt data1
override =gt true
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
=== Check Alarm Criteria ===
Standard Key Words
if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
=== Alarm Exclusions ===
if [message] =~ (i)sometexthere
mutate remove_tag =gt servicename_alarm
Throttle alarms so we do not hit PD rate limits
Page of 52 71 Page of 52 71
if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 5
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle
after_count =gt 5
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 5
period =gt 1200
key =gt servicename_alarm_warning
add_tag =gt servicename_throttled
Page of 53 71 Page of 53 71
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
output
elasticsearch
hosts =gt localhost
stdout codec =gt rubydebug
if servicename_alarm_err in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
Page of 54 71 Page of 54 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 55 71 Page of 55 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
Page of 56 71 Page of 56 71
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert err message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_error in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
Page of 57 71 Page of 57 71
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 58 71 Page of 58 71
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 59 71 Page of 59 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert error message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
Page of 60 71 Page of 60 71
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
Page of 61 71 Page of 61 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 62 71 Page of 62 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt x
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert fatal message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
Page of 63 71 Page of 63 71
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
Page of 64 71 Page of 64 71
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 65 71 Page of 65 71
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt heavy_exclamation_mark
Page of 66 71 Page of 66 71
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert WARNING message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 67 71 Page of 67 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
Page of 68 71 Page of 68 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 69 71 Page of 69 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
Page of 70 71 Page of 70 71
Page of 71 71 Page of 71 71
13
7 To13 delete13 or13 update13 saved13 searches13 click13 13 manage13 saved13 searches13
13
Use13 the13 edit13 icons13 to13 make13 any13 changes13
Page of 26 71 Page of 26 71
13
Build13 simple13 Kibana13 histograms13 Kibana13 allows13 you13 to13 build13 charts13 and13 dashboards13 based13 on13 the13 search13 capability13 offered13 by13 the13 Discover13 page13 Saved13 searches13 can13 be13 used13 as13 the13 basis13 for13 charts13 and13 dashboards13 built13 using13 those13 charts13 To13 build13 a13 simple13 histogram13
1 Select13 the13 Visualize13 page13
13
2 Select13 Ver0cal13 Bar13 Chart13
13
3 To13 use13 a13 saved13 search13 click13 From13 a13 saved13 search13
Page of 27 71 Page of 27 71
13
4 Select13 a13 previously13 saved13 search13
13
5 Select13 to13 edit13 the13 x-shy‐axis
6 Edit13 the13 fields13 with13 informaon13 about13 the13 aggregaon13 field13 interval13 and13 custom13 label13
Page of 28 71 Page of 28 71
13
Page of 29 71 Page of 29 71
7 Click13 the13 green13 arrow13 The13 histogram13 is13 created13 and13 run13 based13 on13 the13 saved13 search
13
8 To13 save13 the13 histogram13 click13 the13 Save13 Visualiza0on13 icon13 and13 add13 a13 tle13
13
13
9 Click13 Save13
10 To13 view13 the13 visualisaon13 click13 the13 Dashboard13 page13
Page of 30 71 Page of 30 71
13
11 Click13 the13 Add13 Visualiza0on13 icon13
13
12 Select13 the13 visiualizaon13 that13 you13 just13 created13
13
Page of 31 71 Page of 31 71
13
13 To13 save13 the13 dashboard13 click13 the13 Save13 Dashboard13 icon13
Page of 32 71 Page of 32 71
13
14 Add13 a13 tle13 and13 select13 whether13 to13 store13 the13 selected13 me13 frame13 with13 the13 dashboard13
13
15 Click13 Save13
16 You13 can13 edit13 the13 shape13 and13 number13 of13 charts13 and13 save13 the13 dashboard13
17 To13 re-shy‐run13 the13 dashboard13 select13 the13 Dashboard13 page13 and13 click13 the13 Load13 Saved13 Dashboard13 icon13
Page of 33 71 Page of 33 71
13
18 Select13 the13 saved13 dashboard13 from13 the13 list13
13
19 The13 dashboard13 runs13
13
Page of 34 71 Page of 34 71
Page of 35 71 Page of 35 71
Aler0ng13 with13 ELK13 ELK13 examines13 the13 log13 records13 in13 process13 to13 send13 alerts13 to13 third-shy‐party13 alerng13 event13 management13 and13 social13 media13 applicaons13 As13 shown13 previously13 in13 the13 ELK13 architecture13 the13 ELK13 strategy13 for13 alerng13 is13 to13 send13 alerts13 from13 the13 Logstash13 component13 directly13 Therefore13 log13 records13 and13 alerts13 are13 sent13 to13 both13 Elascsearch13 and13 third-shy‐party13 tools13 at13 the13 same13 instant13 using13 the13 output13 secon13 of13 the13 Logstash13 configuraon13 Any13 alerts13 are13 generated13 during13 the13 filter13 secon13 processing13 of13 the13 Logstash13 configuraon13
Create13 alerts13 in13 Logstash13 The13 Logstash13 filter13 secon13 is13 used13 to13 trigger13 alerts13 based13 on13 the13 log13 records13 being13 processed13 For13 example13 log13 records13 can13 be13 checked13 for13 errors13 warnings13 crashed13 or13 stopped13 messages13 and13 generate13 an13 alert13
Here13 is13 an13 example13 Logstash13 filter13 code13
13 if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
The13 specific13 tags13 are13 set13 against13 the13 log13 records13 to13 trigger13 the13 alert13 later13 in13 the13 configuraon13 in13 the13 output13 secon13
Control13 the13 number13 of13 generated13 alerts13 using13 the13 Logstash13 throcle13 plugin13 so13 that13 either13 a13 lot13 of13 instances13 in13 a13 fixed13 me13 period13 are13 required13 before13 an13 alert13 is13 sent13 or13 a13 specific13 number13 of13 instances13 triggers13 the13 alert13
Page of 36 71 Page of 36 71
Edit13 the13 plugin13 parameters13 to13 meet13 system13 requirements13
13 13 13 13 13 if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 10
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle13
after_count =gt 20
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 30
period =gt 1200
key =gt servicename_alarm_warning
Page of 37 71 Page of 37 71
add_tag =gt servicename_throttled
13 13 13 13 13 13 13 13 13
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 40
period =gt 1500
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
Enrich13 alerts13 with13 message13 detail13 Some13 of13 the13 third-shy‐party13 integraons13 benefit13 from13 having13 components13 of13 the13 raw13 log13 record13 sent13 in13 addion13 to13 the13 raw13 record13 or13 fields13 in13 the13 record13 edited13 transformed13 or13 translated13
Grok13 Logstash13 allows13 the13 log13 record13 to13 be13 broken13 up13 into13 logical13 secons13 and13 each13 secon13 stored13 in13 a13 new13 parameter13 using13 the13 grok13 parameter13 in13 the13 filter13 secon13 Here13 is13 an13 example13 grok13
13 grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
To13 build13 the13 grok13 statement13 Logstash13 offers13 an13 online13 tool13 at13 hcpsgrokdebugherokuappcom13
Translate13 If13 specific13 fields13 in13 the13 log13 record13 are13 coded13 (for13 example13 the13 Bluemix13 app13 name13 is13 oaen13 not13 sent13 in13 the13 syslog13 messages13 instead13 the13 GUID13 is13 sent13 which13 is13 not13 useful)13 then13 the13 filter13 secon13 translate13 plugin13 can13 be13 used13 to13 add13 new13 fields13 by13 holding13 the13 decoded13 versions13 of13 those13 values13
13 translate
field =gt data1
destination =gt data1
override =gt true
Page of 38 71 Page of 38 71
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
dc50f6e4-c052-436a-bf27-95ba392c4228 monverifyservicemanagement
0c677e82-5bfc-4e38-9155-e416249e3ee6 ms-catalog-postpyretic-pomiculture
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
However13 the13 translate13 plugin13 is13 not13 included13 with13 the13 Logstash13 installaon13 In13 the13 next13 secon13 I13 will13 13 describe13 how13 to13 add13 new13 plugins13 to13 the13 Logstash13 instance13
Add13 custom13 plugins13 to13 Logstash13 As13 an13 open13 source13 tool13 Logstash13 encourages13 users13 to13 develop13 content13 for13 all13 users13 to13 share13 Some13 of13 the13 plugins13 that13 were13 created13 by13 our13 community13 include13 a13 translaon13 plugin13 an13 output13 Slack13 plugin13 and13 an13 output13 syslog13 plugin13 Letrsquos13 look13 at13 how13 to13 add13 these13 plugins13 to13 your13 environment13
Filter13 translate13 plugin13 To13 add13 the13 translate13 plugin13
1 Download13 the13 plugin13
2 Copy13 the13 plugin13 file13 logstash-shy‐filter-shy‐translate-shy‐214gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-filter-translate 214
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Output13 Slack13 plugin13 To13 add13 the13 Slack13 plugin13
Page of 39 71 Page of 39 71
1 Download13 the13 plugin13 13
2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐slack-shy‐014gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-output-slack 014
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Output13 syslog13 plugin13 To13 add13 the13 syslog13 plugin13
1 Download13 the13 plugin13 13
2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐syslog-shy‐214gem13 to13 the13 ELK13 Virtual13 Server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-output-syslog 214
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Integrate13 Logstash13 with13 Slack13 To13 send13 generated13 alerts13 to13 Slack13 complete13 these13 steps13
1 Create13 a13 Slack13 integraon13 13
2 Use13 the13 Slack13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 Slack13 integraon13
113 Login13 to13 Slack13 and13 go13 to13 the13 Apps13 page13 13
Page of 40 71 Page of 40 71
13
313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13
13
A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13
13
Page of 42 71 Page of 42 71
The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13
13
The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13
13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
slack
url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5
channel =gt cloud-operations-mon
username =gt nick_cawood
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13
Page of 43 71 Page of 43 71
Page of 44 71 Page of 44 71
In13 Slack13 the13 alerts13 look13 like13 this13
13
Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13
1 Create13 a13 PagerDuty13 integraon13 13
2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13
1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13
13
2 Click13 Add13 New13 Service13
13
313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13
Page of 45 71 Page of 45 71
13
413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13
513 Click13 Add13 Service13
13
The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13
Page of 46 71 Page of 46 71
13
The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt be7eb1ea9c224018923e818d3e891d0b
incident_key =gt logstashhosttype
Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13
Page of 47 71 Page of 47 71
13
Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13
1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13
2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13
3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13
4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13
13 13 http
url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo
http_method =gt post
format =gt json
For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13
13 13 syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
Page of 48 71 Page of 48 71
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13
13
The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13
if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2
This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13
Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13
Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13
Logstash13 severity Shown13 in13 Omnibus13 as
alert Indeterminate
crical Warning
debug 7
emergency Clear
error Minor
informaonal 6
noce Crical
warning Major
Page of 49 71 Page of 49 71
13
Page of 50 71 Page of 50 71
Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13
input
tcp
port =gt 5000
type =gt syslog
filter
mutate
add_field =gt format =gt cf
grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
translate
field =gt data1
destination =gt data1
override =gt true
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
=== Check Alarm Criteria ===
Standard Key Words
if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
=== Alarm Exclusions ===
if [message] =~ (i)sometexthere
mutate remove_tag =gt servicename_alarm
Throttle alarms so we do not hit PD rate limits
Page of 52 71 Page of 52 71
if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 5
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle
after_count =gt 5
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 5
period =gt 1200
key =gt servicename_alarm_warning
add_tag =gt servicename_throttled
Page of 53 71 Page of 53 71
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
output
elasticsearch
hosts =gt localhost
stdout codec =gt rubydebug
if servicename_alarm_err in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
Page of 54 71 Page of 54 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 55 71 Page of 55 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
Page of 56 71 Page of 56 71
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert err message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_error in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
Page of 57 71 Page of 57 71
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 58 71 Page of 58 71
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 59 71 Page of 59 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert error message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
Page of 60 71 Page of 60 71
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
Page of 61 71 Page of 61 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 62 71 Page of 62 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt x
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert fatal message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
Page of 63 71 Page of 63 71
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
Page of 64 71 Page of 64 71
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 65 71 Page of 65 71
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt heavy_exclamation_mark
Page of 66 71 Page of 66 71
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert WARNING message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 67 71 Page of 67 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
Page of 68 71 Page of 68 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 69 71 Page of 69 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
Page of 70 71 Page of 70 71
Page of 71 71 Page of 71 71
13
Build13 simple13 Kibana13 histograms13 Kibana13 allows13 you13 to13 build13 charts13 and13 dashboards13 based13 on13 the13 search13 capability13 offered13 by13 the13 Discover13 page13 Saved13 searches13 can13 be13 used13 as13 the13 basis13 for13 charts13 and13 dashboards13 built13 using13 those13 charts13 To13 build13 a13 simple13 histogram13
1 Select13 the13 Visualize13 page13
13
2 Select13 Ver0cal13 Bar13 Chart13
13
3 To13 use13 a13 saved13 search13 click13 From13 a13 saved13 search13
Page of 27 71 Page of 27 71
13
4 Select13 a13 previously13 saved13 search13
13
5 Select13 to13 edit13 the13 x-shy‐axis
6 Edit13 the13 fields13 with13 informaon13 about13 the13 aggregaon13 field13 interval13 and13 custom13 label13
Page of 28 71 Page of 28 71
13
Page of 29 71 Page of 29 71
7 Click13 the13 green13 arrow13 The13 histogram13 is13 created13 and13 run13 based13 on13 the13 saved13 search
13
8 To13 save13 the13 histogram13 click13 the13 Save13 Visualiza0on13 icon13 and13 add13 a13 tle13
13
13
9 Click13 Save13
10 To13 view13 the13 visualisaon13 click13 the13 Dashboard13 page13
Page of 30 71 Page of 30 71
13
11 Click13 the13 Add13 Visualiza0on13 icon13
13
12 Select13 the13 visiualizaon13 that13 you13 just13 created13
13
Page of 31 71 Page of 31 71
13
13 To13 save13 the13 dashboard13 click13 the13 Save13 Dashboard13 icon13
Page of 32 71 Page of 32 71
13
14 Add13 a13 tle13 and13 select13 whether13 to13 store13 the13 selected13 me13 frame13 with13 the13 dashboard13
13
15 Click13 Save13
16 You13 can13 edit13 the13 shape13 and13 number13 of13 charts13 and13 save13 the13 dashboard13
17 To13 re-shy‐run13 the13 dashboard13 select13 the13 Dashboard13 page13 and13 click13 the13 Load13 Saved13 Dashboard13 icon13
Page of 33 71 Page of 33 71
13
18 Select13 the13 saved13 dashboard13 from13 the13 list13
13
19 The13 dashboard13 runs13
13
Page of 34 71 Page of 34 71
Page of 35 71 Page of 35 71
Aler0ng13 with13 ELK13 ELK13 examines13 the13 log13 records13 in13 process13 to13 send13 alerts13 to13 third-shy‐party13 alerng13 event13 management13 and13 social13 media13 applicaons13 As13 shown13 previously13 in13 the13 ELK13 architecture13 the13 ELK13 strategy13 for13 alerng13 is13 to13 send13 alerts13 from13 the13 Logstash13 component13 directly13 Therefore13 log13 records13 and13 alerts13 are13 sent13 to13 both13 Elascsearch13 and13 third-shy‐party13 tools13 at13 the13 same13 instant13 using13 the13 output13 secon13 of13 the13 Logstash13 configuraon13 Any13 alerts13 are13 generated13 during13 the13 filter13 secon13 processing13 of13 the13 Logstash13 configuraon13
Create13 alerts13 in13 Logstash13 The13 Logstash13 filter13 secon13 is13 used13 to13 trigger13 alerts13 based13 on13 the13 log13 records13 being13 processed13 For13 example13 log13 records13 can13 be13 checked13 for13 errors13 warnings13 crashed13 or13 stopped13 messages13 and13 generate13 an13 alert13
Here13 is13 an13 example13 Logstash13 filter13 code13
13 if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
The13 specific13 tags13 are13 set13 against13 the13 log13 records13 to13 trigger13 the13 alert13 later13 in13 the13 configuraon13 in13 the13 output13 secon13
Control13 the13 number13 of13 generated13 alerts13 using13 the13 Logstash13 throcle13 plugin13 so13 that13 either13 a13 lot13 of13 instances13 in13 a13 fixed13 me13 period13 are13 required13 before13 an13 alert13 is13 sent13 or13 a13 specific13 number13 of13 instances13 triggers13 the13 alert13
Page of 36 71 Page of 36 71
Edit13 the13 plugin13 parameters13 to13 meet13 system13 requirements13
13 13 13 13 13 if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 10
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle13
after_count =gt 20
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 30
period =gt 1200
key =gt servicename_alarm_warning
Page of 37 71 Page of 37 71
add_tag =gt servicename_throttled
13 13 13 13 13 13 13 13 13
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 40
period =gt 1500
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
Enrich13 alerts13 with13 message13 detail13 Some13 of13 the13 third-shy‐party13 integraons13 benefit13 from13 having13 components13 of13 the13 raw13 log13 record13 sent13 in13 addion13 to13 the13 raw13 record13 or13 fields13 in13 the13 record13 edited13 transformed13 or13 translated13
Grok13 Logstash13 allows13 the13 log13 record13 to13 be13 broken13 up13 into13 logical13 secons13 and13 each13 secon13 stored13 in13 a13 new13 parameter13 using13 the13 grok13 parameter13 in13 the13 filter13 secon13 Here13 is13 an13 example13 grok13
13 grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
To13 build13 the13 grok13 statement13 Logstash13 offers13 an13 online13 tool13 at13 hcpsgrokdebugherokuappcom13
Translate13 If13 specific13 fields13 in13 the13 log13 record13 are13 coded13 (for13 example13 the13 Bluemix13 app13 name13 is13 oaen13 not13 sent13 in13 the13 syslog13 messages13 instead13 the13 GUID13 is13 sent13 which13 is13 not13 useful)13 then13 the13 filter13 secon13 translate13 plugin13 can13 be13 used13 to13 add13 new13 fields13 by13 holding13 the13 decoded13 versions13 of13 those13 values13
13 translate
field =gt data1
destination =gt data1
override =gt true
Page of 38 71 Page of 38 71
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
dc50f6e4-c052-436a-bf27-95ba392c4228 monverifyservicemanagement
0c677e82-5bfc-4e38-9155-e416249e3ee6 ms-catalog-postpyretic-pomiculture
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
However13 the13 translate13 plugin13 is13 not13 included13 with13 the13 Logstash13 installaon13 In13 the13 next13 secon13 I13 will13 13 describe13 how13 to13 add13 new13 plugins13 to13 the13 Logstash13 instance13
Add13 custom13 plugins13 to13 Logstash13 As13 an13 open13 source13 tool13 Logstash13 encourages13 users13 to13 develop13 content13 for13 all13 users13 to13 share13 Some13 of13 the13 plugins13 that13 were13 created13 by13 our13 community13 include13 a13 translaon13 plugin13 an13 output13 Slack13 plugin13 and13 an13 output13 syslog13 plugin13 Letrsquos13 look13 at13 how13 to13 add13 these13 plugins13 to13 your13 environment13
Filter13 translate13 plugin13 To13 add13 the13 translate13 plugin13
1 Download13 the13 plugin13
2 Copy13 the13 plugin13 file13 logstash-shy‐filter-shy‐translate-shy‐214gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-filter-translate 214
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Output13 Slack13 plugin13 To13 add13 the13 Slack13 plugin13
Page of 39 71 Page of 39 71
1 Download13 the13 plugin13 13
2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐slack-shy‐014gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-output-slack 014
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Output13 syslog13 plugin13 To13 add13 the13 syslog13 plugin13
1 Download13 the13 plugin13 13
2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐syslog-shy‐214gem13 to13 the13 ELK13 Virtual13 Server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-output-syslog 214
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Integrate13 Logstash13 with13 Slack13 To13 send13 generated13 alerts13 to13 Slack13 complete13 these13 steps13
1 Create13 a13 Slack13 integraon13 13
2 Use13 the13 Slack13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 Slack13 integraon13
113 Login13 to13 Slack13 and13 go13 to13 the13 Apps13 page13 13
Page of 40 71 Page of 40 71
13
313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13
13
A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13
13
Page of 42 71 Page of 42 71
The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13
13
The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13
13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
slack
url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5
channel =gt cloud-operations-mon
username =gt nick_cawood
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13
Page of 43 71 Page of 43 71
Page of 44 71 Page of 44 71
In13 Slack13 the13 alerts13 look13 like13 this13
13
Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13
1 Create13 a13 PagerDuty13 integraon13 13
2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13
1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13
13
2 Click13 Add13 New13 Service13
13
313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13
Page of 45 71 Page of 45 71
13
413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13
513 Click13 Add13 Service13
13
The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13
Page of 46 71 Page of 46 71
13
The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt be7eb1ea9c224018923e818d3e891d0b
incident_key =gt logstashhosttype
Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13
Page of 47 71 Page of 47 71
13
Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13
1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13
2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13
3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13
4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13
13 13 http
url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo
http_method =gt post
format =gt json
For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13
13 13 syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
Page of 48 71 Page of 48 71
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13
13
The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13
if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2
This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13
Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13
Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13
Logstash13 severity Shown13 in13 Omnibus13 as
alert Indeterminate
crical Warning
debug 7
emergency Clear
error Minor
informaonal 6
noce Crical
warning Major
Page of 49 71 Page of 49 71
13
Page of 50 71 Page of 50 71
Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13
input
tcp
port =gt 5000
type =gt syslog
filter
mutate
add_field =gt format =gt cf
grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
translate
field =gt data1
destination =gt data1
override =gt true
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
=== Check Alarm Criteria ===
Standard Key Words
if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
=== Alarm Exclusions ===
if [message] =~ (i)sometexthere
mutate remove_tag =gt servicename_alarm
Throttle alarms so we do not hit PD rate limits
Page of 52 71 Page of 52 71
if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 5
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle
after_count =gt 5
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 5
period =gt 1200
key =gt servicename_alarm_warning
add_tag =gt servicename_throttled
Page of 53 71 Page of 53 71
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
output
elasticsearch
hosts =gt localhost
stdout codec =gt rubydebug
if servicename_alarm_err in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
Page of 54 71 Page of 54 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 55 71 Page of 55 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
Page of 56 71 Page of 56 71
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert err message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_error in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
Page of 57 71 Page of 57 71
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 58 71 Page of 58 71
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 59 71 Page of 59 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert error message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
Page of 60 71 Page of 60 71
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
Page of 61 71 Page of 61 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 62 71 Page of 62 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt x
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert fatal message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
Page of 63 71 Page of 63 71
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
Page of 64 71 Page of 64 71
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 65 71 Page of 65 71
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt heavy_exclamation_mark
Page of 66 71 Page of 66 71
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert WARNING message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 67 71 Page of 67 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
Page of 68 71 Page of 68 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 69 71 Page of 69 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
Page of 70 71 Page of 70 71
Page of 71 71 Page of 71 71
13
4 Select13 a13 previously13 saved13 search13
13
5 Select13 to13 edit13 the13 x-shy‐axis
6 Edit13 the13 fields13 with13 informaon13 about13 the13 aggregaon13 field13 interval13 and13 custom13 label13
Page of 28 71 Page of 28 71
13
Page of 29 71 Page of 29 71
7 Click13 the13 green13 arrow13 The13 histogram13 is13 created13 and13 run13 based13 on13 the13 saved13 search
13
8 To13 save13 the13 histogram13 click13 the13 Save13 Visualiza0on13 icon13 and13 add13 a13 tle13
13
13
9 Click13 Save13
10 To13 view13 the13 visualisaon13 click13 the13 Dashboard13 page13
Page of 30 71 Page of 30 71
13
11 Click13 the13 Add13 Visualiza0on13 icon13
13
12 Select13 the13 visiualizaon13 that13 you13 just13 created13
13
Page of 31 71 Page of 31 71
13
13 To13 save13 the13 dashboard13 click13 the13 Save13 Dashboard13 icon13
Page of 32 71 Page of 32 71
13
14 Add13 a13 tle13 and13 select13 whether13 to13 store13 the13 selected13 me13 frame13 with13 the13 dashboard13
13
15 Click13 Save13
16 You13 can13 edit13 the13 shape13 and13 number13 of13 charts13 and13 save13 the13 dashboard13
17 To13 re-shy‐run13 the13 dashboard13 select13 the13 Dashboard13 page13 and13 click13 the13 Load13 Saved13 Dashboard13 icon13
Page of 33 71 Page of 33 71
13
18 Select13 the13 saved13 dashboard13 from13 the13 list13
13
19 The13 dashboard13 runs13
13
Page of 34 71 Page of 34 71
Page of 35 71 Page of 35 71
Aler0ng13 with13 ELK13 ELK13 examines13 the13 log13 records13 in13 process13 to13 send13 alerts13 to13 third-shy‐party13 alerng13 event13 management13 and13 social13 media13 applicaons13 As13 shown13 previously13 in13 the13 ELK13 architecture13 the13 ELK13 strategy13 for13 alerng13 is13 to13 send13 alerts13 from13 the13 Logstash13 component13 directly13 Therefore13 log13 records13 and13 alerts13 are13 sent13 to13 both13 Elascsearch13 and13 third-shy‐party13 tools13 at13 the13 same13 instant13 using13 the13 output13 secon13 of13 the13 Logstash13 configuraon13 Any13 alerts13 are13 generated13 during13 the13 filter13 secon13 processing13 of13 the13 Logstash13 configuraon13
Create13 alerts13 in13 Logstash13 The13 Logstash13 filter13 secon13 is13 used13 to13 trigger13 alerts13 based13 on13 the13 log13 records13 being13 processed13 For13 example13 log13 records13 can13 be13 checked13 for13 errors13 warnings13 crashed13 or13 stopped13 messages13 and13 generate13 an13 alert13
Here13 is13 an13 example13 Logstash13 filter13 code13
13 if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
The13 specific13 tags13 are13 set13 against13 the13 log13 records13 to13 trigger13 the13 alert13 later13 in13 the13 configuraon13 in13 the13 output13 secon13
Control13 the13 number13 of13 generated13 alerts13 using13 the13 Logstash13 throcle13 plugin13 so13 that13 either13 a13 lot13 of13 instances13 in13 a13 fixed13 me13 period13 are13 required13 before13 an13 alert13 is13 sent13 or13 a13 specific13 number13 of13 instances13 triggers13 the13 alert13
Page of 36 71 Page of 36 71
Edit13 the13 plugin13 parameters13 to13 meet13 system13 requirements13
13 13 13 13 13 if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 10
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle13
after_count =gt 20
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 30
period =gt 1200
key =gt servicename_alarm_warning
Page of 37 71 Page of 37 71
add_tag =gt servicename_throttled
13 13 13 13 13 13 13 13 13
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 40
period =gt 1500
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
Enrich13 alerts13 with13 message13 detail13 Some13 of13 the13 third-shy‐party13 integraons13 benefit13 from13 having13 components13 of13 the13 raw13 log13 record13 sent13 in13 addion13 to13 the13 raw13 record13 or13 fields13 in13 the13 record13 edited13 transformed13 or13 translated13
Grok13 Logstash13 allows13 the13 log13 record13 to13 be13 broken13 up13 into13 logical13 secons13 and13 each13 secon13 stored13 in13 a13 new13 parameter13 using13 the13 grok13 parameter13 in13 the13 filter13 secon13 Here13 is13 an13 example13 grok13
13 grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
To13 build13 the13 grok13 statement13 Logstash13 offers13 an13 online13 tool13 at13 hcpsgrokdebugherokuappcom13
Translate13 If13 specific13 fields13 in13 the13 log13 record13 are13 coded13 (for13 example13 the13 Bluemix13 app13 name13 is13 oaen13 not13 sent13 in13 the13 syslog13 messages13 instead13 the13 GUID13 is13 sent13 which13 is13 not13 useful)13 then13 the13 filter13 secon13 translate13 plugin13 can13 be13 used13 to13 add13 new13 fields13 by13 holding13 the13 decoded13 versions13 of13 those13 values13
13 translate
field =gt data1
destination =gt data1
override =gt true
Page of 38 71 Page of 38 71
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
dc50f6e4-c052-436a-bf27-95ba392c4228 monverifyservicemanagement
0c677e82-5bfc-4e38-9155-e416249e3ee6 ms-catalog-postpyretic-pomiculture
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
However13 the13 translate13 plugin13 is13 not13 included13 with13 the13 Logstash13 installaon13 In13 the13 next13 secon13 I13 will13 13 describe13 how13 to13 add13 new13 plugins13 to13 the13 Logstash13 instance13
Add13 custom13 plugins13 to13 Logstash13 As13 an13 open13 source13 tool13 Logstash13 encourages13 users13 to13 develop13 content13 for13 all13 users13 to13 share13 Some13 of13 the13 plugins13 that13 were13 created13 by13 our13 community13 include13 a13 translaon13 plugin13 an13 output13 Slack13 plugin13 and13 an13 output13 syslog13 plugin13 Letrsquos13 look13 at13 how13 to13 add13 these13 plugins13 to13 your13 environment13
Filter13 translate13 plugin13 To13 add13 the13 translate13 plugin13
1 Download13 the13 plugin13
2 Copy13 the13 plugin13 file13 logstash-shy‐filter-shy‐translate-shy‐214gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-filter-translate 214
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Output13 Slack13 plugin13 To13 add13 the13 Slack13 plugin13
Page of 39 71 Page of 39 71
1 Download13 the13 plugin13 13
2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐slack-shy‐014gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-output-slack 014
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Output13 syslog13 plugin13 To13 add13 the13 syslog13 plugin13
1 Download13 the13 plugin13 13
2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐syslog-shy‐214gem13 to13 the13 ELK13 Virtual13 Server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-output-syslog 214
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Integrate13 Logstash13 with13 Slack13 To13 send13 generated13 alerts13 to13 Slack13 complete13 these13 steps13
1 Create13 a13 Slack13 integraon13 13
2 Use13 the13 Slack13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 Slack13 integraon13
113 Login13 to13 Slack13 and13 go13 to13 the13 Apps13 page13 13
Page of 40 71 Page of 40 71
13
313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13
13
A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13
13
Page of 42 71 Page of 42 71
The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13
13
The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13
13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
slack
url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5
channel =gt cloud-operations-mon
username =gt nick_cawood
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13
Page of 43 71 Page of 43 71
Page of 44 71 Page of 44 71
In13 Slack13 the13 alerts13 look13 like13 this13
13
Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13
1 Create13 a13 PagerDuty13 integraon13 13
2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13
1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13
13
2 Click13 Add13 New13 Service13
13
313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13
Page of 45 71 Page of 45 71
13
413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13
513 Click13 Add13 Service13
13
The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13
Page of 46 71 Page of 46 71
13
The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt be7eb1ea9c224018923e818d3e891d0b
incident_key =gt logstashhosttype
Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13
Page of 47 71 Page of 47 71
13
Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13
1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13
2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13
3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13
4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13
13 13 http
url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo
http_method =gt post
format =gt json
For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13
13 13 syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
Page of 48 71 Page of 48 71
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13
13
The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13
if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2
This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13
Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13
Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13
Logstash13 severity Shown13 in13 Omnibus13 as
alert Indeterminate
crical Warning
debug 7
emergency Clear
error Minor
informaonal 6
noce Crical
warning Major
Page of 49 71 Page of 49 71
13
Page of 50 71 Page of 50 71
Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13
input
tcp
port =gt 5000
type =gt syslog
filter
mutate
add_field =gt format =gt cf
grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
translate
field =gt data1
destination =gt data1
override =gt true
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
=== Check Alarm Criteria ===
Standard Key Words
if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
=== Alarm Exclusions ===
if [message] =~ (i)sometexthere
mutate remove_tag =gt servicename_alarm
Throttle alarms so we do not hit PD rate limits
Page of 52 71 Page of 52 71
if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 5
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle
after_count =gt 5
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 5
period =gt 1200
key =gt servicename_alarm_warning
add_tag =gt servicename_throttled
Page of 53 71 Page of 53 71
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
output
elasticsearch
hosts =gt localhost
stdout codec =gt rubydebug
if servicename_alarm_err in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
Page of 54 71 Page of 54 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 55 71 Page of 55 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
Page of 56 71 Page of 56 71
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert err message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_error in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
Page of 57 71 Page of 57 71
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 58 71 Page of 58 71
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 59 71 Page of 59 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert error message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
Page of 60 71 Page of 60 71
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
Page of 61 71 Page of 61 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 62 71 Page of 62 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt x
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert fatal message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
Page of 63 71 Page of 63 71
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
Page of 64 71 Page of 64 71
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 65 71 Page of 65 71
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt heavy_exclamation_mark
Page of 66 71 Page of 66 71
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert WARNING message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 67 71 Page of 67 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
Page of 68 71 Page of 68 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 69 71 Page of 69 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
Page of 70 71 Page of 70 71
Page of 71 71 Page of 71 71
13
Page of 29 71 Page of 29 71
7 Click13 the13 green13 arrow13 The13 histogram13 is13 created13 and13 run13 based13 on13 the13 saved13 search
13
8 To13 save13 the13 histogram13 click13 the13 Save13 Visualiza0on13 icon13 and13 add13 a13 tle13
13
13
9 Click13 Save13
10 To13 view13 the13 visualisaon13 click13 the13 Dashboard13 page13
Page of 30 71 Page of 30 71
13
11 Click13 the13 Add13 Visualiza0on13 icon13
13
12 Select13 the13 visiualizaon13 that13 you13 just13 created13
13
Page of 31 71 Page of 31 71
13
13 To13 save13 the13 dashboard13 click13 the13 Save13 Dashboard13 icon13
Page of 32 71 Page of 32 71
13
14 Add13 a13 tle13 and13 select13 whether13 to13 store13 the13 selected13 me13 frame13 with13 the13 dashboard13
13
15 Click13 Save13
16 You13 can13 edit13 the13 shape13 and13 number13 of13 charts13 and13 save13 the13 dashboard13
17 To13 re-shy‐run13 the13 dashboard13 select13 the13 Dashboard13 page13 and13 click13 the13 Load13 Saved13 Dashboard13 icon13
Page of 33 71 Page of 33 71
13
18 Select13 the13 saved13 dashboard13 from13 the13 list13
13
19 The13 dashboard13 runs13
13
Page of 34 71 Page of 34 71
Page of 35 71 Page of 35 71
Aler0ng13 with13 ELK13 ELK13 examines13 the13 log13 records13 in13 process13 to13 send13 alerts13 to13 third-shy‐party13 alerng13 event13 management13 and13 social13 media13 applicaons13 As13 shown13 previously13 in13 the13 ELK13 architecture13 the13 ELK13 strategy13 for13 alerng13 is13 to13 send13 alerts13 from13 the13 Logstash13 component13 directly13 Therefore13 log13 records13 and13 alerts13 are13 sent13 to13 both13 Elascsearch13 and13 third-shy‐party13 tools13 at13 the13 same13 instant13 using13 the13 output13 secon13 of13 the13 Logstash13 configuraon13 Any13 alerts13 are13 generated13 during13 the13 filter13 secon13 processing13 of13 the13 Logstash13 configuraon13
Create13 alerts13 in13 Logstash13 The13 Logstash13 filter13 secon13 is13 used13 to13 trigger13 alerts13 based13 on13 the13 log13 records13 being13 processed13 For13 example13 log13 records13 can13 be13 checked13 for13 errors13 warnings13 crashed13 or13 stopped13 messages13 and13 generate13 an13 alert13
Here13 is13 an13 example13 Logstash13 filter13 code13
13 if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
The13 specific13 tags13 are13 set13 against13 the13 log13 records13 to13 trigger13 the13 alert13 later13 in13 the13 configuraon13 in13 the13 output13 secon13
Control13 the13 number13 of13 generated13 alerts13 using13 the13 Logstash13 throcle13 plugin13 so13 that13 either13 a13 lot13 of13 instances13 in13 a13 fixed13 me13 period13 are13 required13 before13 an13 alert13 is13 sent13 or13 a13 specific13 number13 of13 instances13 triggers13 the13 alert13
Page of 36 71 Page of 36 71
Edit13 the13 plugin13 parameters13 to13 meet13 system13 requirements13
13 13 13 13 13 if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 10
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle13
after_count =gt 20
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 30
period =gt 1200
key =gt servicename_alarm_warning
Page of 37 71 Page of 37 71
add_tag =gt servicename_throttled
13 13 13 13 13 13 13 13 13
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 40
period =gt 1500
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
Enrich13 alerts13 with13 message13 detail13 Some13 of13 the13 third-shy‐party13 integraons13 benefit13 from13 having13 components13 of13 the13 raw13 log13 record13 sent13 in13 addion13 to13 the13 raw13 record13 or13 fields13 in13 the13 record13 edited13 transformed13 or13 translated13
Grok13 Logstash13 allows13 the13 log13 record13 to13 be13 broken13 up13 into13 logical13 secons13 and13 each13 secon13 stored13 in13 a13 new13 parameter13 using13 the13 grok13 parameter13 in13 the13 filter13 secon13 Here13 is13 an13 example13 grok13
13 grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
To13 build13 the13 grok13 statement13 Logstash13 offers13 an13 online13 tool13 at13 hcpsgrokdebugherokuappcom13
Translate13 If13 specific13 fields13 in13 the13 log13 record13 are13 coded13 (for13 example13 the13 Bluemix13 app13 name13 is13 oaen13 not13 sent13 in13 the13 syslog13 messages13 instead13 the13 GUID13 is13 sent13 which13 is13 not13 useful)13 then13 the13 filter13 secon13 translate13 plugin13 can13 be13 used13 to13 add13 new13 fields13 by13 holding13 the13 decoded13 versions13 of13 those13 values13
13 translate
field =gt data1
destination =gt data1
override =gt true
Page of 38 71 Page of 38 71
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
dc50f6e4-c052-436a-bf27-95ba392c4228 monverifyservicemanagement
0c677e82-5bfc-4e38-9155-e416249e3ee6 ms-catalog-postpyretic-pomiculture
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
However13 the13 translate13 plugin13 is13 not13 included13 with13 the13 Logstash13 installaon13 In13 the13 next13 secon13 I13 will13 13 describe13 how13 to13 add13 new13 plugins13 to13 the13 Logstash13 instance13
Add13 custom13 plugins13 to13 Logstash13 As13 an13 open13 source13 tool13 Logstash13 encourages13 users13 to13 develop13 content13 for13 all13 users13 to13 share13 Some13 of13 the13 plugins13 that13 were13 created13 by13 our13 community13 include13 a13 translaon13 plugin13 an13 output13 Slack13 plugin13 and13 an13 output13 syslog13 plugin13 Letrsquos13 look13 at13 how13 to13 add13 these13 plugins13 to13 your13 environment13
Filter13 translate13 plugin13 To13 add13 the13 translate13 plugin13
1 Download13 the13 plugin13
2 Copy13 the13 plugin13 file13 logstash-shy‐filter-shy‐translate-shy‐214gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-filter-translate 214
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Output13 Slack13 plugin13 To13 add13 the13 Slack13 plugin13
Page of 39 71 Page of 39 71
1 Download13 the13 plugin13 13
2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐slack-shy‐014gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-output-slack 014
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Output13 syslog13 plugin13 To13 add13 the13 syslog13 plugin13
1 Download13 the13 plugin13 13
2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐syslog-shy‐214gem13 to13 the13 ELK13 Virtual13 Server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-output-syslog 214
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Integrate13 Logstash13 with13 Slack13 To13 send13 generated13 alerts13 to13 Slack13 complete13 these13 steps13
1 Create13 a13 Slack13 integraon13 13
2 Use13 the13 Slack13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 Slack13 integraon13
113 Login13 to13 Slack13 and13 go13 to13 the13 Apps13 page13 13
Page of 40 71 Page of 40 71
13
313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13
13
A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13
13
Page of 42 71 Page of 42 71
The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13
13
The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13
13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
slack
url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5
channel =gt cloud-operations-mon
username =gt nick_cawood
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13
Page of 43 71 Page of 43 71
Page of 44 71 Page of 44 71
In13 Slack13 the13 alerts13 look13 like13 this13
13
Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13
1 Create13 a13 PagerDuty13 integraon13 13
2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13
1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13
13
2 Click13 Add13 New13 Service13
13
313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13
Page of 45 71 Page of 45 71
13
413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13
513 Click13 Add13 Service13
13
The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13
Page of 46 71 Page of 46 71
13
The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt be7eb1ea9c224018923e818d3e891d0b
incident_key =gt logstashhosttype
Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13
Page of 47 71 Page of 47 71
13
Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13
1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13
2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13
3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13
4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13
13 13 http
url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo
http_method =gt post
format =gt json
For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13
13 13 syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
Page of 48 71 Page of 48 71
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13
13
The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13
if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2
This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13
Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13
Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13
Logstash13 severity Shown13 in13 Omnibus13 as
alert Indeterminate
crical Warning
debug 7
emergency Clear
error Minor
informaonal 6
noce Crical
warning Major
Page of 49 71 Page of 49 71
13
Page of 50 71 Page of 50 71
Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13
input
tcp
port =gt 5000
type =gt syslog
filter
mutate
add_field =gt format =gt cf
grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
translate
field =gt data1
destination =gt data1
override =gt true
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
=== Check Alarm Criteria ===
Standard Key Words
if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
=== Alarm Exclusions ===
if [message] =~ (i)sometexthere
mutate remove_tag =gt servicename_alarm
Throttle alarms so we do not hit PD rate limits
Page of 52 71 Page of 52 71
if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 5
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle
after_count =gt 5
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 5
period =gt 1200
key =gt servicename_alarm_warning
add_tag =gt servicename_throttled
Page of 53 71 Page of 53 71
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
output
elasticsearch
hosts =gt localhost
stdout codec =gt rubydebug
if servicename_alarm_err in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
Page of 54 71 Page of 54 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 55 71 Page of 55 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
Page of 56 71 Page of 56 71
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert err message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_error in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
Page of 57 71 Page of 57 71
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 58 71 Page of 58 71
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 59 71 Page of 59 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert error message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
Page of 60 71 Page of 60 71
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
Page of 61 71 Page of 61 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 62 71 Page of 62 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt x
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert fatal message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
Page of 63 71 Page of 63 71
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
Page of 64 71 Page of 64 71
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 65 71 Page of 65 71
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt heavy_exclamation_mark
Page of 66 71 Page of 66 71
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert WARNING message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 67 71 Page of 67 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
Page of 68 71 Page of 68 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 69 71 Page of 69 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
Page of 70 71 Page of 70 71
Page of 71 71 Page of 71 71
7 Click13 the13 green13 arrow13 The13 histogram13 is13 created13 and13 run13 based13 on13 the13 saved13 search
13
8 To13 save13 the13 histogram13 click13 the13 Save13 Visualiza0on13 icon13 and13 add13 a13 tle13
13
13
9 Click13 Save13
10 To13 view13 the13 visualisaon13 click13 the13 Dashboard13 page13
Page of 30 71 Page of 30 71
13
11 Click13 the13 Add13 Visualiza0on13 icon13
13
12 Select13 the13 visiualizaon13 that13 you13 just13 created13
13
Page of 31 71 Page of 31 71
13
13 To13 save13 the13 dashboard13 click13 the13 Save13 Dashboard13 icon13
Page of 32 71 Page of 32 71
13
14 Add13 a13 tle13 and13 select13 whether13 to13 store13 the13 selected13 me13 frame13 with13 the13 dashboard13
13
15 Click13 Save13
16 You13 can13 edit13 the13 shape13 and13 number13 of13 charts13 and13 save13 the13 dashboard13
17 To13 re-shy‐run13 the13 dashboard13 select13 the13 Dashboard13 page13 and13 click13 the13 Load13 Saved13 Dashboard13 icon13
Page of 33 71 Page of 33 71
13
18 Select13 the13 saved13 dashboard13 from13 the13 list13
13
19 The13 dashboard13 runs13
13
Page of 34 71 Page of 34 71
Page of 35 71 Page of 35 71
Aler0ng13 with13 ELK13 ELK13 examines13 the13 log13 records13 in13 process13 to13 send13 alerts13 to13 third-shy‐party13 alerng13 event13 management13 and13 social13 media13 applicaons13 As13 shown13 previously13 in13 the13 ELK13 architecture13 the13 ELK13 strategy13 for13 alerng13 is13 to13 send13 alerts13 from13 the13 Logstash13 component13 directly13 Therefore13 log13 records13 and13 alerts13 are13 sent13 to13 both13 Elascsearch13 and13 third-shy‐party13 tools13 at13 the13 same13 instant13 using13 the13 output13 secon13 of13 the13 Logstash13 configuraon13 Any13 alerts13 are13 generated13 during13 the13 filter13 secon13 processing13 of13 the13 Logstash13 configuraon13
Create13 alerts13 in13 Logstash13 The13 Logstash13 filter13 secon13 is13 used13 to13 trigger13 alerts13 based13 on13 the13 log13 records13 being13 processed13 For13 example13 log13 records13 can13 be13 checked13 for13 errors13 warnings13 crashed13 or13 stopped13 messages13 and13 generate13 an13 alert13
Here13 is13 an13 example13 Logstash13 filter13 code13
13 if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
The13 specific13 tags13 are13 set13 against13 the13 log13 records13 to13 trigger13 the13 alert13 later13 in13 the13 configuraon13 in13 the13 output13 secon13
Control13 the13 number13 of13 generated13 alerts13 using13 the13 Logstash13 throcle13 plugin13 so13 that13 either13 a13 lot13 of13 instances13 in13 a13 fixed13 me13 period13 are13 required13 before13 an13 alert13 is13 sent13 or13 a13 specific13 number13 of13 instances13 triggers13 the13 alert13
Page of 36 71 Page of 36 71
Edit13 the13 plugin13 parameters13 to13 meet13 system13 requirements13
13 13 13 13 13 if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 10
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle13
after_count =gt 20
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 30
period =gt 1200
key =gt servicename_alarm_warning
Page of 37 71 Page of 37 71
add_tag =gt servicename_throttled
13 13 13 13 13 13 13 13 13
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 40
period =gt 1500
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
Enrich13 alerts13 with13 message13 detail13 Some13 of13 the13 third-shy‐party13 integraons13 benefit13 from13 having13 components13 of13 the13 raw13 log13 record13 sent13 in13 addion13 to13 the13 raw13 record13 or13 fields13 in13 the13 record13 edited13 transformed13 or13 translated13
Grok13 Logstash13 allows13 the13 log13 record13 to13 be13 broken13 up13 into13 logical13 secons13 and13 each13 secon13 stored13 in13 a13 new13 parameter13 using13 the13 grok13 parameter13 in13 the13 filter13 secon13 Here13 is13 an13 example13 grok13
13 grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
To13 build13 the13 grok13 statement13 Logstash13 offers13 an13 online13 tool13 at13 hcpsgrokdebugherokuappcom13
Translate13 If13 specific13 fields13 in13 the13 log13 record13 are13 coded13 (for13 example13 the13 Bluemix13 app13 name13 is13 oaen13 not13 sent13 in13 the13 syslog13 messages13 instead13 the13 GUID13 is13 sent13 which13 is13 not13 useful)13 then13 the13 filter13 secon13 translate13 plugin13 can13 be13 used13 to13 add13 new13 fields13 by13 holding13 the13 decoded13 versions13 of13 those13 values13
13 translate
field =gt data1
destination =gt data1
override =gt true
Page of 38 71 Page of 38 71
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
dc50f6e4-c052-436a-bf27-95ba392c4228 monverifyservicemanagement
0c677e82-5bfc-4e38-9155-e416249e3ee6 ms-catalog-postpyretic-pomiculture
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
However13 the13 translate13 plugin13 is13 not13 included13 with13 the13 Logstash13 installaon13 In13 the13 next13 secon13 I13 will13 13 describe13 how13 to13 add13 new13 plugins13 to13 the13 Logstash13 instance13
Add13 custom13 plugins13 to13 Logstash13 As13 an13 open13 source13 tool13 Logstash13 encourages13 users13 to13 develop13 content13 for13 all13 users13 to13 share13 Some13 of13 the13 plugins13 that13 were13 created13 by13 our13 community13 include13 a13 translaon13 plugin13 an13 output13 Slack13 plugin13 and13 an13 output13 syslog13 plugin13 Letrsquos13 look13 at13 how13 to13 add13 these13 plugins13 to13 your13 environment13
Filter13 translate13 plugin13 To13 add13 the13 translate13 plugin13
1 Download13 the13 plugin13
2 Copy13 the13 plugin13 file13 logstash-shy‐filter-shy‐translate-shy‐214gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-filter-translate 214
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Output13 Slack13 plugin13 To13 add13 the13 Slack13 plugin13
Page of 39 71 Page of 39 71
1 Download13 the13 plugin13 13
2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐slack-shy‐014gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-output-slack 014
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Output13 syslog13 plugin13 To13 add13 the13 syslog13 plugin13
1 Download13 the13 plugin13 13
2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐syslog-shy‐214gem13 to13 the13 ELK13 Virtual13 Server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-output-syslog 214
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Integrate13 Logstash13 with13 Slack13 To13 send13 generated13 alerts13 to13 Slack13 complete13 these13 steps13
1 Create13 a13 Slack13 integraon13 13
2 Use13 the13 Slack13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 Slack13 integraon13
113 Login13 to13 Slack13 and13 go13 to13 the13 Apps13 page13 13
Page of 40 71 Page of 40 71
13
313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13
13
A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13
13
Page of 42 71 Page of 42 71
The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13
13
The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13
13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
slack
url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5
channel =gt cloud-operations-mon
username =gt nick_cawood
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13
Page of 43 71 Page of 43 71
Page of 44 71 Page of 44 71
In13 Slack13 the13 alerts13 look13 like13 this13
13
Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13
1 Create13 a13 PagerDuty13 integraon13 13
2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13
1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13
13
2 Click13 Add13 New13 Service13
13
313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13
Page of 45 71 Page of 45 71
13
413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13
513 Click13 Add13 Service13
13
The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13
Page of 46 71 Page of 46 71
13
The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt be7eb1ea9c224018923e818d3e891d0b
incident_key =gt logstashhosttype
Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13
Page of 47 71 Page of 47 71
13
Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13
1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13
2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13
3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13
4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13
13 13 http
url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo
http_method =gt post
format =gt json
For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13
13 13 syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
Page of 48 71 Page of 48 71
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13
13
The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13
if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2
This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13
Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13
Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13
Logstash13 severity Shown13 in13 Omnibus13 as
alert Indeterminate
crical Warning
debug 7
emergency Clear
error Minor
informaonal 6
noce Crical
warning Major
Page of 49 71 Page of 49 71
13
Page of 50 71 Page of 50 71
Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13
input
tcp
port =gt 5000
type =gt syslog
filter
mutate
add_field =gt format =gt cf
grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
translate
field =gt data1
destination =gt data1
override =gt true
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
=== Check Alarm Criteria ===
Standard Key Words
if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
=== Alarm Exclusions ===
if [message] =~ (i)sometexthere
mutate remove_tag =gt servicename_alarm
Throttle alarms so we do not hit PD rate limits
Page of 52 71 Page of 52 71
if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 5
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle
after_count =gt 5
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 5
period =gt 1200
key =gt servicename_alarm_warning
add_tag =gt servicename_throttled
Page of 53 71 Page of 53 71
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
output
elasticsearch
hosts =gt localhost
stdout codec =gt rubydebug
if servicename_alarm_err in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
Page of 54 71 Page of 54 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 55 71 Page of 55 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
Page of 56 71 Page of 56 71
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert err message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_error in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
Page of 57 71 Page of 57 71
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 58 71 Page of 58 71
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 59 71 Page of 59 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert error message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
Page of 60 71 Page of 60 71
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
Page of 61 71 Page of 61 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 62 71 Page of 62 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt x
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert fatal message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
Page of 63 71 Page of 63 71
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
Page of 64 71 Page of 64 71
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 65 71 Page of 65 71
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt heavy_exclamation_mark
Page of 66 71 Page of 66 71
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert WARNING message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 67 71 Page of 67 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
Page of 68 71 Page of 68 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 69 71 Page of 69 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
Page of 70 71 Page of 70 71
Page of 71 71 Page of 71 71
13
11 Click13 the13 Add13 Visualiza0on13 icon13
13
12 Select13 the13 visiualizaon13 that13 you13 just13 created13
13
Page of 31 71 Page of 31 71
13
13 To13 save13 the13 dashboard13 click13 the13 Save13 Dashboard13 icon13
Page of 32 71 Page of 32 71
13
14 Add13 a13 tle13 and13 select13 whether13 to13 store13 the13 selected13 me13 frame13 with13 the13 dashboard13
13
15 Click13 Save13
16 You13 can13 edit13 the13 shape13 and13 number13 of13 charts13 and13 save13 the13 dashboard13
17 To13 re-shy‐run13 the13 dashboard13 select13 the13 Dashboard13 page13 and13 click13 the13 Load13 Saved13 Dashboard13 icon13
Page of 33 71 Page of 33 71
13
18 Select13 the13 saved13 dashboard13 from13 the13 list13
13
19 The13 dashboard13 runs13
13
Page of 34 71 Page of 34 71
Page of 35 71 Page of 35 71
Aler0ng13 with13 ELK13 ELK13 examines13 the13 log13 records13 in13 process13 to13 send13 alerts13 to13 third-shy‐party13 alerng13 event13 management13 and13 social13 media13 applicaons13 As13 shown13 previously13 in13 the13 ELK13 architecture13 the13 ELK13 strategy13 for13 alerng13 is13 to13 send13 alerts13 from13 the13 Logstash13 component13 directly13 Therefore13 log13 records13 and13 alerts13 are13 sent13 to13 both13 Elascsearch13 and13 third-shy‐party13 tools13 at13 the13 same13 instant13 using13 the13 output13 secon13 of13 the13 Logstash13 configuraon13 Any13 alerts13 are13 generated13 during13 the13 filter13 secon13 processing13 of13 the13 Logstash13 configuraon13
Create13 alerts13 in13 Logstash13 The13 Logstash13 filter13 secon13 is13 used13 to13 trigger13 alerts13 based13 on13 the13 log13 records13 being13 processed13 For13 example13 log13 records13 can13 be13 checked13 for13 errors13 warnings13 crashed13 or13 stopped13 messages13 and13 generate13 an13 alert13
Here13 is13 an13 example13 Logstash13 filter13 code13
13 if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
The13 specific13 tags13 are13 set13 against13 the13 log13 records13 to13 trigger13 the13 alert13 later13 in13 the13 configuraon13 in13 the13 output13 secon13
Control13 the13 number13 of13 generated13 alerts13 using13 the13 Logstash13 throcle13 plugin13 so13 that13 either13 a13 lot13 of13 instances13 in13 a13 fixed13 me13 period13 are13 required13 before13 an13 alert13 is13 sent13 or13 a13 specific13 number13 of13 instances13 triggers13 the13 alert13
Page of 36 71 Page of 36 71
Edit13 the13 plugin13 parameters13 to13 meet13 system13 requirements13
13 13 13 13 13 if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 10
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle13
after_count =gt 20
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 30
period =gt 1200
key =gt servicename_alarm_warning
Page of 37 71 Page of 37 71
add_tag =gt servicename_throttled
13 13 13 13 13 13 13 13 13
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 40
period =gt 1500
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
Enrich13 alerts13 with13 message13 detail13 Some13 of13 the13 third-shy‐party13 integraons13 benefit13 from13 having13 components13 of13 the13 raw13 log13 record13 sent13 in13 addion13 to13 the13 raw13 record13 or13 fields13 in13 the13 record13 edited13 transformed13 or13 translated13
Grok13 Logstash13 allows13 the13 log13 record13 to13 be13 broken13 up13 into13 logical13 secons13 and13 each13 secon13 stored13 in13 a13 new13 parameter13 using13 the13 grok13 parameter13 in13 the13 filter13 secon13 Here13 is13 an13 example13 grok13
13 grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
To13 build13 the13 grok13 statement13 Logstash13 offers13 an13 online13 tool13 at13 hcpsgrokdebugherokuappcom13
Translate13 If13 specific13 fields13 in13 the13 log13 record13 are13 coded13 (for13 example13 the13 Bluemix13 app13 name13 is13 oaen13 not13 sent13 in13 the13 syslog13 messages13 instead13 the13 GUID13 is13 sent13 which13 is13 not13 useful)13 then13 the13 filter13 secon13 translate13 plugin13 can13 be13 used13 to13 add13 new13 fields13 by13 holding13 the13 decoded13 versions13 of13 those13 values13
13 translate
field =gt data1
destination =gt data1
override =gt true
Page of 38 71 Page of 38 71
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
dc50f6e4-c052-436a-bf27-95ba392c4228 monverifyservicemanagement
0c677e82-5bfc-4e38-9155-e416249e3ee6 ms-catalog-postpyretic-pomiculture
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
However13 the13 translate13 plugin13 is13 not13 included13 with13 the13 Logstash13 installaon13 In13 the13 next13 secon13 I13 will13 13 describe13 how13 to13 add13 new13 plugins13 to13 the13 Logstash13 instance13
Add13 custom13 plugins13 to13 Logstash13 As13 an13 open13 source13 tool13 Logstash13 encourages13 users13 to13 develop13 content13 for13 all13 users13 to13 share13 Some13 of13 the13 plugins13 that13 were13 created13 by13 our13 community13 include13 a13 translaon13 plugin13 an13 output13 Slack13 plugin13 and13 an13 output13 syslog13 plugin13 Letrsquos13 look13 at13 how13 to13 add13 these13 plugins13 to13 your13 environment13
Filter13 translate13 plugin13 To13 add13 the13 translate13 plugin13
1 Download13 the13 plugin13
2 Copy13 the13 plugin13 file13 logstash-shy‐filter-shy‐translate-shy‐214gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-filter-translate 214
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Output13 Slack13 plugin13 To13 add13 the13 Slack13 plugin13
Page of 39 71 Page of 39 71
1 Download13 the13 plugin13 13
2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐slack-shy‐014gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-output-slack 014
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Output13 syslog13 plugin13 To13 add13 the13 syslog13 plugin13
1 Download13 the13 plugin13 13
2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐syslog-shy‐214gem13 to13 the13 ELK13 Virtual13 Server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-output-syslog 214
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Integrate13 Logstash13 with13 Slack13 To13 send13 generated13 alerts13 to13 Slack13 complete13 these13 steps13
1 Create13 a13 Slack13 integraon13 13
2 Use13 the13 Slack13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 Slack13 integraon13
113 Login13 to13 Slack13 and13 go13 to13 the13 Apps13 page13 13
Page of 40 71 Page of 40 71
13
313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13
13
A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13
13
Page of 42 71 Page of 42 71
The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13
13
The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13
13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
slack
url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5
channel =gt cloud-operations-mon
username =gt nick_cawood
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13
Page of 43 71 Page of 43 71
Page of 44 71 Page of 44 71
In13 Slack13 the13 alerts13 look13 like13 this13
13
Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13
1 Create13 a13 PagerDuty13 integraon13 13
2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13
1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13
13
2 Click13 Add13 New13 Service13
13
313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13
Page of 45 71 Page of 45 71
13
413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13
513 Click13 Add13 Service13
13
The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13
Page of 46 71 Page of 46 71
13
The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt be7eb1ea9c224018923e818d3e891d0b
incident_key =gt logstashhosttype
Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13
Page of 47 71 Page of 47 71
13
Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13
1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13
2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13
3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13
4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13
13 13 http
url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo
http_method =gt post
format =gt json
For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13
13 13 syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
Page of 48 71 Page of 48 71
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13
13
The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13
if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2
This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13
Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13
Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13
Logstash13 severity Shown13 in13 Omnibus13 as
alert Indeterminate
crical Warning
debug 7
emergency Clear
error Minor
informaonal 6
noce Crical
warning Major
Page of 49 71 Page of 49 71
13
Page of 50 71 Page of 50 71
Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13
input
tcp
port =gt 5000
type =gt syslog
filter
mutate
add_field =gt format =gt cf
grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
translate
field =gt data1
destination =gt data1
override =gt true
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
=== Check Alarm Criteria ===
Standard Key Words
if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
=== Alarm Exclusions ===
if [message] =~ (i)sometexthere
mutate remove_tag =gt servicename_alarm
Throttle alarms so we do not hit PD rate limits
Page of 52 71 Page of 52 71
if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 5
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle
after_count =gt 5
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 5
period =gt 1200
key =gt servicename_alarm_warning
add_tag =gt servicename_throttled
Page of 53 71 Page of 53 71
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
output
elasticsearch
hosts =gt localhost
stdout codec =gt rubydebug
if servicename_alarm_err in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
Page of 54 71 Page of 54 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 55 71 Page of 55 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
Page of 56 71 Page of 56 71
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert err message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_error in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
Page of 57 71 Page of 57 71
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 58 71 Page of 58 71
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 59 71 Page of 59 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert error message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
Page of 60 71 Page of 60 71
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
Page of 61 71 Page of 61 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 62 71 Page of 62 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt x
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert fatal message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
Page of 63 71 Page of 63 71
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
Page of 64 71 Page of 64 71
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 65 71 Page of 65 71
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt heavy_exclamation_mark
Page of 66 71 Page of 66 71
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert WARNING message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 67 71 Page of 67 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
Page of 68 71 Page of 68 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 69 71 Page of 69 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
Page of 70 71 Page of 70 71
Page of 71 71 Page of 71 71
13
13 To13 save13 the13 dashboard13 click13 the13 Save13 Dashboard13 icon13
Page of 32 71 Page of 32 71
13
14 Add13 a13 tle13 and13 select13 whether13 to13 store13 the13 selected13 me13 frame13 with13 the13 dashboard13
13
15 Click13 Save13
16 You13 can13 edit13 the13 shape13 and13 number13 of13 charts13 and13 save13 the13 dashboard13
17 To13 re-shy‐run13 the13 dashboard13 select13 the13 Dashboard13 page13 and13 click13 the13 Load13 Saved13 Dashboard13 icon13
Page of 33 71 Page of 33 71
13
18 Select13 the13 saved13 dashboard13 from13 the13 list13
13
19 The13 dashboard13 runs13
13
Page of 34 71 Page of 34 71
Page of 35 71 Page of 35 71
Aler0ng13 with13 ELK13 ELK13 examines13 the13 log13 records13 in13 process13 to13 send13 alerts13 to13 third-shy‐party13 alerng13 event13 management13 and13 social13 media13 applicaons13 As13 shown13 previously13 in13 the13 ELK13 architecture13 the13 ELK13 strategy13 for13 alerng13 is13 to13 send13 alerts13 from13 the13 Logstash13 component13 directly13 Therefore13 log13 records13 and13 alerts13 are13 sent13 to13 both13 Elascsearch13 and13 third-shy‐party13 tools13 at13 the13 same13 instant13 using13 the13 output13 secon13 of13 the13 Logstash13 configuraon13 Any13 alerts13 are13 generated13 during13 the13 filter13 secon13 processing13 of13 the13 Logstash13 configuraon13
Create13 alerts13 in13 Logstash13 The13 Logstash13 filter13 secon13 is13 used13 to13 trigger13 alerts13 based13 on13 the13 log13 records13 being13 processed13 For13 example13 log13 records13 can13 be13 checked13 for13 errors13 warnings13 crashed13 or13 stopped13 messages13 and13 generate13 an13 alert13
Here13 is13 an13 example13 Logstash13 filter13 code13
13 if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
The13 specific13 tags13 are13 set13 against13 the13 log13 records13 to13 trigger13 the13 alert13 later13 in13 the13 configuraon13 in13 the13 output13 secon13
Control13 the13 number13 of13 generated13 alerts13 using13 the13 Logstash13 throcle13 plugin13 so13 that13 either13 a13 lot13 of13 instances13 in13 a13 fixed13 me13 period13 are13 required13 before13 an13 alert13 is13 sent13 or13 a13 specific13 number13 of13 instances13 triggers13 the13 alert13
Page of 36 71 Page of 36 71
Edit13 the13 plugin13 parameters13 to13 meet13 system13 requirements13
13 13 13 13 13 if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 10
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle13
after_count =gt 20
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 30
period =gt 1200
key =gt servicename_alarm_warning
Page of 37 71 Page of 37 71
add_tag =gt servicename_throttled
13 13 13 13 13 13 13 13 13
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 40
period =gt 1500
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
Enrich13 alerts13 with13 message13 detail13 Some13 of13 the13 third-shy‐party13 integraons13 benefit13 from13 having13 components13 of13 the13 raw13 log13 record13 sent13 in13 addion13 to13 the13 raw13 record13 or13 fields13 in13 the13 record13 edited13 transformed13 or13 translated13
Grok13 Logstash13 allows13 the13 log13 record13 to13 be13 broken13 up13 into13 logical13 secons13 and13 each13 secon13 stored13 in13 a13 new13 parameter13 using13 the13 grok13 parameter13 in13 the13 filter13 secon13 Here13 is13 an13 example13 grok13
13 grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
To13 build13 the13 grok13 statement13 Logstash13 offers13 an13 online13 tool13 at13 hcpsgrokdebugherokuappcom13
Translate13 If13 specific13 fields13 in13 the13 log13 record13 are13 coded13 (for13 example13 the13 Bluemix13 app13 name13 is13 oaen13 not13 sent13 in13 the13 syslog13 messages13 instead13 the13 GUID13 is13 sent13 which13 is13 not13 useful)13 then13 the13 filter13 secon13 translate13 plugin13 can13 be13 used13 to13 add13 new13 fields13 by13 holding13 the13 decoded13 versions13 of13 those13 values13
13 translate
field =gt data1
destination =gt data1
override =gt true
Page of 38 71 Page of 38 71
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
dc50f6e4-c052-436a-bf27-95ba392c4228 monverifyservicemanagement
0c677e82-5bfc-4e38-9155-e416249e3ee6 ms-catalog-postpyretic-pomiculture
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
However13 the13 translate13 plugin13 is13 not13 included13 with13 the13 Logstash13 installaon13 In13 the13 next13 secon13 I13 will13 13 describe13 how13 to13 add13 new13 plugins13 to13 the13 Logstash13 instance13
Add13 custom13 plugins13 to13 Logstash13 As13 an13 open13 source13 tool13 Logstash13 encourages13 users13 to13 develop13 content13 for13 all13 users13 to13 share13 Some13 of13 the13 plugins13 that13 were13 created13 by13 our13 community13 include13 a13 translaon13 plugin13 an13 output13 Slack13 plugin13 and13 an13 output13 syslog13 plugin13 Letrsquos13 look13 at13 how13 to13 add13 these13 plugins13 to13 your13 environment13
Filter13 translate13 plugin13 To13 add13 the13 translate13 plugin13
1 Download13 the13 plugin13
2 Copy13 the13 plugin13 file13 logstash-shy‐filter-shy‐translate-shy‐214gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-filter-translate 214
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Output13 Slack13 plugin13 To13 add13 the13 Slack13 plugin13
Page of 39 71 Page of 39 71
1 Download13 the13 plugin13 13
2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐slack-shy‐014gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-output-slack 014
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Output13 syslog13 plugin13 To13 add13 the13 syslog13 plugin13
1 Download13 the13 plugin13 13
2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐syslog-shy‐214gem13 to13 the13 ELK13 Virtual13 Server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-output-syslog 214
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Integrate13 Logstash13 with13 Slack13 To13 send13 generated13 alerts13 to13 Slack13 complete13 these13 steps13
1 Create13 a13 Slack13 integraon13 13
2 Use13 the13 Slack13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 Slack13 integraon13
113 Login13 to13 Slack13 and13 go13 to13 the13 Apps13 page13 13
Page of 40 71 Page of 40 71
13
313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13
13
A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13
13
Page of 42 71 Page of 42 71
The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13
13
The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13
13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
slack
url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5
channel =gt cloud-operations-mon
username =gt nick_cawood
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13
Page of 43 71 Page of 43 71
Page of 44 71 Page of 44 71
In13 Slack13 the13 alerts13 look13 like13 this13
13
Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13
1 Create13 a13 PagerDuty13 integraon13 13
2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13
1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13
13
2 Click13 Add13 New13 Service13
13
313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13
Page of 45 71 Page of 45 71
13
413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13
513 Click13 Add13 Service13
13
The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13
Page of 46 71 Page of 46 71
13
The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt be7eb1ea9c224018923e818d3e891d0b
incident_key =gt logstashhosttype
Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13
Page of 47 71 Page of 47 71
13
Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13
1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13
2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13
3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13
4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13
13 13 http
url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo
http_method =gt post
format =gt json
For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13
13 13 syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
Page of 48 71 Page of 48 71
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13
13
The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13
if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2
This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13
Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13
Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13
Logstash13 severity Shown13 in13 Omnibus13 as
alert Indeterminate
crical Warning
debug 7
emergency Clear
error Minor
informaonal 6
noce Crical
warning Major
Page of 49 71 Page of 49 71
13
Page of 50 71 Page of 50 71
Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13
input
tcp
port =gt 5000
type =gt syslog
filter
mutate
add_field =gt format =gt cf
grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
translate
field =gt data1
destination =gt data1
override =gt true
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
=== Check Alarm Criteria ===
Standard Key Words
if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
=== Alarm Exclusions ===
if [message] =~ (i)sometexthere
mutate remove_tag =gt servicename_alarm
Throttle alarms so we do not hit PD rate limits
Page of 52 71 Page of 52 71
if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 5
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle
after_count =gt 5
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 5
period =gt 1200
key =gt servicename_alarm_warning
add_tag =gt servicename_throttled
Page of 53 71 Page of 53 71
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
output
elasticsearch
hosts =gt localhost
stdout codec =gt rubydebug
if servicename_alarm_err in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
Page of 54 71 Page of 54 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 55 71 Page of 55 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
Page of 56 71 Page of 56 71
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert err message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_error in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
Page of 57 71 Page of 57 71
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 58 71 Page of 58 71
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 59 71 Page of 59 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert error message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
Page of 60 71 Page of 60 71
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
Page of 61 71 Page of 61 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 62 71 Page of 62 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt x
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert fatal message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
Page of 63 71 Page of 63 71
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
Page of 64 71 Page of 64 71
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 65 71 Page of 65 71
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt heavy_exclamation_mark
Page of 66 71 Page of 66 71
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert WARNING message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 67 71 Page of 67 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
Page of 68 71 Page of 68 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 69 71 Page of 69 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
Page of 70 71 Page of 70 71
Page of 71 71 Page of 71 71
13
14 Add13 a13 tle13 and13 select13 whether13 to13 store13 the13 selected13 me13 frame13 with13 the13 dashboard13
13
15 Click13 Save13
16 You13 can13 edit13 the13 shape13 and13 number13 of13 charts13 and13 save13 the13 dashboard13
17 To13 re-shy‐run13 the13 dashboard13 select13 the13 Dashboard13 page13 and13 click13 the13 Load13 Saved13 Dashboard13 icon13
Page of 33 71 Page of 33 71
13
18 Select13 the13 saved13 dashboard13 from13 the13 list13
13
19 The13 dashboard13 runs13
13
Page of 34 71 Page of 34 71
Page of 35 71 Page of 35 71
Aler0ng13 with13 ELK13 ELK13 examines13 the13 log13 records13 in13 process13 to13 send13 alerts13 to13 third-shy‐party13 alerng13 event13 management13 and13 social13 media13 applicaons13 As13 shown13 previously13 in13 the13 ELK13 architecture13 the13 ELK13 strategy13 for13 alerng13 is13 to13 send13 alerts13 from13 the13 Logstash13 component13 directly13 Therefore13 log13 records13 and13 alerts13 are13 sent13 to13 both13 Elascsearch13 and13 third-shy‐party13 tools13 at13 the13 same13 instant13 using13 the13 output13 secon13 of13 the13 Logstash13 configuraon13 Any13 alerts13 are13 generated13 during13 the13 filter13 secon13 processing13 of13 the13 Logstash13 configuraon13
Create13 alerts13 in13 Logstash13 The13 Logstash13 filter13 secon13 is13 used13 to13 trigger13 alerts13 based13 on13 the13 log13 records13 being13 processed13 For13 example13 log13 records13 can13 be13 checked13 for13 errors13 warnings13 crashed13 or13 stopped13 messages13 and13 generate13 an13 alert13
Here13 is13 an13 example13 Logstash13 filter13 code13
13 if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
The13 specific13 tags13 are13 set13 against13 the13 log13 records13 to13 trigger13 the13 alert13 later13 in13 the13 configuraon13 in13 the13 output13 secon13
Control13 the13 number13 of13 generated13 alerts13 using13 the13 Logstash13 throcle13 plugin13 so13 that13 either13 a13 lot13 of13 instances13 in13 a13 fixed13 me13 period13 are13 required13 before13 an13 alert13 is13 sent13 or13 a13 specific13 number13 of13 instances13 triggers13 the13 alert13
Page of 36 71 Page of 36 71
Edit13 the13 plugin13 parameters13 to13 meet13 system13 requirements13
13 13 13 13 13 if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 10
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle13
after_count =gt 20
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 30
period =gt 1200
key =gt servicename_alarm_warning
Page of 37 71 Page of 37 71
add_tag =gt servicename_throttled
13 13 13 13 13 13 13 13 13
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 40
period =gt 1500
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
Enrich13 alerts13 with13 message13 detail13 Some13 of13 the13 third-shy‐party13 integraons13 benefit13 from13 having13 components13 of13 the13 raw13 log13 record13 sent13 in13 addion13 to13 the13 raw13 record13 or13 fields13 in13 the13 record13 edited13 transformed13 or13 translated13
Grok13 Logstash13 allows13 the13 log13 record13 to13 be13 broken13 up13 into13 logical13 secons13 and13 each13 secon13 stored13 in13 a13 new13 parameter13 using13 the13 grok13 parameter13 in13 the13 filter13 secon13 Here13 is13 an13 example13 grok13
13 grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
To13 build13 the13 grok13 statement13 Logstash13 offers13 an13 online13 tool13 at13 hcpsgrokdebugherokuappcom13
Translate13 If13 specific13 fields13 in13 the13 log13 record13 are13 coded13 (for13 example13 the13 Bluemix13 app13 name13 is13 oaen13 not13 sent13 in13 the13 syslog13 messages13 instead13 the13 GUID13 is13 sent13 which13 is13 not13 useful)13 then13 the13 filter13 secon13 translate13 plugin13 can13 be13 used13 to13 add13 new13 fields13 by13 holding13 the13 decoded13 versions13 of13 those13 values13
13 translate
field =gt data1
destination =gt data1
override =gt true
Page of 38 71 Page of 38 71
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
dc50f6e4-c052-436a-bf27-95ba392c4228 monverifyservicemanagement
0c677e82-5bfc-4e38-9155-e416249e3ee6 ms-catalog-postpyretic-pomiculture
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
However13 the13 translate13 plugin13 is13 not13 included13 with13 the13 Logstash13 installaon13 In13 the13 next13 secon13 I13 will13 13 describe13 how13 to13 add13 new13 plugins13 to13 the13 Logstash13 instance13
Add13 custom13 plugins13 to13 Logstash13 As13 an13 open13 source13 tool13 Logstash13 encourages13 users13 to13 develop13 content13 for13 all13 users13 to13 share13 Some13 of13 the13 plugins13 that13 were13 created13 by13 our13 community13 include13 a13 translaon13 plugin13 an13 output13 Slack13 plugin13 and13 an13 output13 syslog13 plugin13 Letrsquos13 look13 at13 how13 to13 add13 these13 plugins13 to13 your13 environment13
Filter13 translate13 plugin13 To13 add13 the13 translate13 plugin13
1 Download13 the13 plugin13
2 Copy13 the13 plugin13 file13 logstash-shy‐filter-shy‐translate-shy‐214gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-filter-translate 214
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Output13 Slack13 plugin13 To13 add13 the13 Slack13 plugin13
Page of 39 71 Page of 39 71
1 Download13 the13 plugin13 13
2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐slack-shy‐014gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-output-slack 014
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Output13 syslog13 plugin13 To13 add13 the13 syslog13 plugin13
1 Download13 the13 plugin13 13
2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐syslog-shy‐214gem13 to13 the13 ELK13 Virtual13 Server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-output-syslog 214
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Integrate13 Logstash13 with13 Slack13 To13 send13 generated13 alerts13 to13 Slack13 complete13 these13 steps13
1 Create13 a13 Slack13 integraon13 13
2 Use13 the13 Slack13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 Slack13 integraon13
113 Login13 to13 Slack13 and13 go13 to13 the13 Apps13 page13 13
Page of 40 71 Page of 40 71
13
313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13
13
A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13
13
Page of 42 71 Page of 42 71
The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13
13
The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13
13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
slack
url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5
channel =gt cloud-operations-mon
username =gt nick_cawood
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13
Page of 43 71 Page of 43 71
Page of 44 71 Page of 44 71
In13 Slack13 the13 alerts13 look13 like13 this13
13
Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13
1 Create13 a13 PagerDuty13 integraon13 13
2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13
1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13
13
2 Click13 Add13 New13 Service13
13
313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13
Page of 45 71 Page of 45 71
13
413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13
513 Click13 Add13 Service13
13
The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13
Page of 46 71 Page of 46 71
13
The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt be7eb1ea9c224018923e818d3e891d0b
incident_key =gt logstashhosttype
Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13
Page of 47 71 Page of 47 71
13
Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13
1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13
2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13
3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13
4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13
13 13 http
url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo
http_method =gt post
format =gt json
For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13
13 13 syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
Page of 48 71 Page of 48 71
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13
13
The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13
if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2
This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13
Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13
Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13
Logstash13 severity Shown13 in13 Omnibus13 as
alert Indeterminate
crical Warning
debug 7
emergency Clear
error Minor
informaonal 6
noce Crical
warning Major
Page of 49 71 Page of 49 71
13
Page of 50 71 Page of 50 71
Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13
input
tcp
port =gt 5000
type =gt syslog
filter
mutate
add_field =gt format =gt cf
grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
translate
field =gt data1
destination =gt data1
override =gt true
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
=== Check Alarm Criteria ===
Standard Key Words
if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
=== Alarm Exclusions ===
if [message] =~ (i)sometexthere
mutate remove_tag =gt servicename_alarm
Throttle alarms so we do not hit PD rate limits
Page of 52 71 Page of 52 71
if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 5
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle
after_count =gt 5
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 5
period =gt 1200
key =gt servicename_alarm_warning
add_tag =gt servicename_throttled
Page of 53 71 Page of 53 71
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
output
elasticsearch
hosts =gt localhost
stdout codec =gt rubydebug
if servicename_alarm_err in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
Page of 54 71 Page of 54 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 55 71 Page of 55 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
Page of 56 71 Page of 56 71
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert err message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_error in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
Page of 57 71 Page of 57 71
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 58 71 Page of 58 71
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 59 71 Page of 59 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert error message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
Page of 60 71 Page of 60 71
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
Page of 61 71 Page of 61 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 62 71 Page of 62 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt x
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert fatal message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
Page of 63 71 Page of 63 71
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
Page of 64 71 Page of 64 71
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 65 71 Page of 65 71
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt heavy_exclamation_mark
Page of 66 71 Page of 66 71
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert WARNING message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 67 71 Page of 67 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
Page of 68 71 Page of 68 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 69 71 Page of 69 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
Page of 70 71 Page of 70 71
Page of 71 71 Page of 71 71
13
18 Select13 the13 saved13 dashboard13 from13 the13 list13
13
19 The13 dashboard13 runs13
13
Page of 34 71 Page of 34 71
Page of 35 71 Page of 35 71
Aler0ng13 with13 ELK13 ELK13 examines13 the13 log13 records13 in13 process13 to13 send13 alerts13 to13 third-shy‐party13 alerng13 event13 management13 and13 social13 media13 applicaons13 As13 shown13 previously13 in13 the13 ELK13 architecture13 the13 ELK13 strategy13 for13 alerng13 is13 to13 send13 alerts13 from13 the13 Logstash13 component13 directly13 Therefore13 log13 records13 and13 alerts13 are13 sent13 to13 both13 Elascsearch13 and13 third-shy‐party13 tools13 at13 the13 same13 instant13 using13 the13 output13 secon13 of13 the13 Logstash13 configuraon13 Any13 alerts13 are13 generated13 during13 the13 filter13 secon13 processing13 of13 the13 Logstash13 configuraon13
Create13 alerts13 in13 Logstash13 The13 Logstash13 filter13 secon13 is13 used13 to13 trigger13 alerts13 based13 on13 the13 log13 records13 being13 processed13 For13 example13 log13 records13 can13 be13 checked13 for13 errors13 warnings13 crashed13 or13 stopped13 messages13 and13 generate13 an13 alert13
Here13 is13 an13 example13 Logstash13 filter13 code13
13 if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
The13 specific13 tags13 are13 set13 against13 the13 log13 records13 to13 trigger13 the13 alert13 later13 in13 the13 configuraon13 in13 the13 output13 secon13
Control13 the13 number13 of13 generated13 alerts13 using13 the13 Logstash13 throcle13 plugin13 so13 that13 either13 a13 lot13 of13 instances13 in13 a13 fixed13 me13 period13 are13 required13 before13 an13 alert13 is13 sent13 or13 a13 specific13 number13 of13 instances13 triggers13 the13 alert13
Page of 36 71 Page of 36 71
Edit13 the13 plugin13 parameters13 to13 meet13 system13 requirements13
13 13 13 13 13 if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 10
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle13
after_count =gt 20
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 30
period =gt 1200
key =gt servicename_alarm_warning
Page of 37 71 Page of 37 71
add_tag =gt servicename_throttled
13 13 13 13 13 13 13 13 13
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 40
period =gt 1500
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
Enrich13 alerts13 with13 message13 detail13 Some13 of13 the13 third-shy‐party13 integraons13 benefit13 from13 having13 components13 of13 the13 raw13 log13 record13 sent13 in13 addion13 to13 the13 raw13 record13 or13 fields13 in13 the13 record13 edited13 transformed13 or13 translated13
Grok13 Logstash13 allows13 the13 log13 record13 to13 be13 broken13 up13 into13 logical13 secons13 and13 each13 secon13 stored13 in13 a13 new13 parameter13 using13 the13 grok13 parameter13 in13 the13 filter13 secon13 Here13 is13 an13 example13 grok13
13 grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
To13 build13 the13 grok13 statement13 Logstash13 offers13 an13 online13 tool13 at13 hcpsgrokdebugherokuappcom13
Translate13 If13 specific13 fields13 in13 the13 log13 record13 are13 coded13 (for13 example13 the13 Bluemix13 app13 name13 is13 oaen13 not13 sent13 in13 the13 syslog13 messages13 instead13 the13 GUID13 is13 sent13 which13 is13 not13 useful)13 then13 the13 filter13 secon13 translate13 plugin13 can13 be13 used13 to13 add13 new13 fields13 by13 holding13 the13 decoded13 versions13 of13 those13 values13
13 translate
field =gt data1
destination =gt data1
override =gt true
Page of 38 71 Page of 38 71
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
dc50f6e4-c052-436a-bf27-95ba392c4228 monverifyservicemanagement
0c677e82-5bfc-4e38-9155-e416249e3ee6 ms-catalog-postpyretic-pomiculture
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
However13 the13 translate13 plugin13 is13 not13 included13 with13 the13 Logstash13 installaon13 In13 the13 next13 secon13 I13 will13 13 describe13 how13 to13 add13 new13 plugins13 to13 the13 Logstash13 instance13
Add13 custom13 plugins13 to13 Logstash13 As13 an13 open13 source13 tool13 Logstash13 encourages13 users13 to13 develop13 content13 for13 all13 users13 to13 share13 Some13 of13 the13 plugins13 that13 were13 created13 by13 our13 community13 include13 a13 translaon13 plugin13 an13 output13 Slack13 plugin13 and13 an13 output13 syslog13 plugin13 Letrsquos13 look13 at13 how13 to13 add13 these13 plugins13 to13 your13 environment13
Filter13 translate13 plugin13 To13 add13 the13 translate13 plugin13
1 Download13 the13 plugin13
2 Copy13 the13 plugin13 file13 logstash-shy‐filter-shy‐translate-shy‐214gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-filter-translate 214
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Output13 Slack13 plugin13 To13 add13 the13 Slack13 plugin13
Page of 39 71 Page of 39 71
1 Download13 the13 plugin13 13
2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐slack-shy‐014gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-output-slack 014
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Output13 syslog13 plugin13 To13 add13 the13 syslog13 plugin13
1 Download13 the13 plugin13 13
2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐syslog-shy‐214gem13 to13 the13 ELK13 Virtual13 Server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-output-syslog 214
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Integrate13 Logstash13 with13 Slack13 To13 send13 generated13 alerts13 to13 Slack13 complete13 these13 steps13
1 Create13 a13 Slack13 integraon13 13
2 Use13 the13 Slack13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 Slack13 integraon13
113 Login13 to13 Slack13 and13 go13 to13 the13 Apps13 page13 13
Page of 40 71 Page of 40 71
13
313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13
13
A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13
13
Page of 42 71 Page of 42 71
The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13
13
The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13
13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
slack
url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5
channel =gt cloud-operations-mon
username =gt nick_cawood
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13
Page of 43 71 Page of 43 71
Page of 44 71 Page of 44 71
In13 Slack13 the13 alerts13 look13 like13 this13
13
Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13
1 Create13 a13 PagerDuty13 integraon13 13
2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13
1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13
13
2 Click13 Add13 New13 Service13
13
313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13
Page of 45 71 Page of 45 71
13
413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13
513 Click13 Add13 Service13
13
The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13
Page of 46 71 Page of 46 71
13
The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt be7eb1ea9c224018923e818d3e891d0b
incident_key =gt logstashhosttype
Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13
Page of 47 71 Page of 47 71
13
Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13
1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13
2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13
3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13
4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13
13 13 http
url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo
http_method =gt post
format =gt json
For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13
13 13 syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
Page of 48 71 Page of 48 71
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13
13
The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13
if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2
This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13
Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13
Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13
Logstash13 severity Shown13 in13 Omnibus13 as
alert Indeterminate
crical Warning
debug 7
emergency Clear
error Minor
informaonal 6
noce Crical
warning Major
Page of 49 71 Page of 49 71
13
Page of 50 71 Page of 50 71
Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13
input
tcp
port =gt 5000
type =gt syslog
filter
mutate
add_field =gt format =gt cf
grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
translate
field =gt data1
destination =gt data1
override =gt true
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
=== Check Alarm Criteria ===
Standard Key Words
if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
=== Alarm Exclusions ===
if [message] =~ (i)sometexthere
mutate remove_tag =gt servicename_alarm
Throttle alarms so we do not hit PD rate limits
Page of 52 71 Page of 52 71
if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 5
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle
after_count =gt 5
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 5
period =gt 1200
key =gt servicename_alarm_warning
add_tag =gt servicename_throttled
Page of 53 71 Page of 53 71
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
output
elasticsearch
hosts =gt localhost
stdout codec =gt rubydebug
if servicename_alarm_err in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
Page of 54 71 Page of 54 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 55 71 Page of 55 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
Page of 56 71 Page of 56 71
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert err message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_error in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
Page of 57 71 Page of 57 71
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 58 71 Page of 58 71
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 59 71 Page of 59 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert error message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
Page of 60 71 Page of 60 71
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
Page of 61 71 Page of 61 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 62 71 Page of 62 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt x
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert fatal message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
Page of 63 71 Page of 63 71
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
Page of 64 71 Page of 64 71
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 65 71 Page of 65 71
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt heavy_exclamation_mark
Page of 66 71 Page of 66 71
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert WARNING message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 67 71 Page of 67 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
Page of 68 71 Page of 68 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 69 71 Page of 69 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
Page of 70 71 Page of 70 71
Page of 71 71 Page of 71 71
Page of 35 71 Page of 35 71
Aler0ng13 with13 ELK13 ELK13 examines13 the13 log13 records13 in13 process13 to13 send13 alerts13 to13 third-shy‐party13 alerng13 event13 management13 and13 social13 media13 applicaons13 As13 shown13 previously13 in13 the13 ELK13 architecture13 the13 ELK13 strategy13 for13 alerng13 is13 to13 send13 alerts13 from13 the13 Logstash13 component13 directly13 Therefore13 log13 records13 and13 alerts13 are13 sent13 to13 both13 Elascsearch13 and13 third-shy‐party13 tools13 at13 the13 same13 instant13 using13 the13 output13 secon13 of13 the13 Logstash13 configuraon13 Any13 alerts13 are13 generated13 during13 the13 filter13 secon13 processing13 of13 the13 Logstash13 configuraon13
Create13 alerts13 in13 Logstash13 The13 Logstash13 filter13 secon13 is13 used13 to13 trigger13 alerts13 based13 on13 the13 log13 records13 being13 processed13 For13 example13 log13 records13 can13 be13 checked13 for13 errors13 warnings13 crashed13 or13 stopped13 messages13 and13 generate13 an13 alert13
Here13 is13 an13 example13 Logstash13 filter13 code13
13 if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
The13 specific13 tags13 are13 set13 against13 the13 log13 records13 to13 trigger13 the13 alert13 later13 in13 the13 configuraon13 in13 the13 output13 secon13
Control13 the13 number13 of13 generated13 alerts13 using13 the13 Logstash13 throcle13 plugin13 so13 that13 either13 a13 lot13 of13 instances13 in13 a13 fixed13 me13 period13 are13 required13 before13 an13 alert13 is13 sent13 or13 a13 specific13 number13 of13 instances13 triggers13 the13 alert13
Page of 36 71 Page of 36 71
Edit13 the13 plugin13 parameters13 to13 meet13 system13 requirements13
13 13 13 13 13 if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 10
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle13
after_count =gt 20
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 30
period =gt 1200
key =gt servicename_alarm_warning
Page of 37 71 Page of 37 71
add_tag =gt servicename_throttled
13 13 13 13 13 13 13 13 13
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 40
period =gt 1500
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
Enrich13 alerts13 with13 message13 detail13 Some13 of13 the13 third-shy‐party13 integraons13 benefit13 from13 having13 components13 of13 the13 raw13 log13 record13 sent13 in13 addion13 to13 the13 raw13 record13 or13 fields13 in13 the13 record13 edited13 transformed13 or13 translated13
Grok13 Logstash13 allows13 the13 log13 record13 to13 be13 broken13 up13 into13 logical13 secons13 and13 each13 secon13 stored13 in13 a13 new13 parameter13 using13 the13 grok13 parameter13 in13 the13 filter13 secon13 Here13 is13 an13 example13 grok13
13 grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
To13 build13 the13 grok13 statement13 Logstash13 offers13 an13 online13 tool13 at13 hcpsgrokdebugherokuappcom13
Translate13 If13 specific13 fields13 in13 the13 log13 record13 are13 coded13 (for13 example13 the13 Bluemix13 app13 name13 is13 oaen13 not13 sent13 in13 the13 syslog13 messages13 instead13 the13 GUID13 is13 sent13 which13 is13 not13 useful)13 then13 the13 filter13 secon13 translate13 plugin13 can13 be13 used13 to13 add13 new13 fields13 by13 holding13 the13 decoded13 versions13 of13 those13 values13
13 translate
field =gt data1
destination =gt data1
override =gt true
Page of 38 71 Page of 38 71
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
dc50f6e4-c052-436a-bf27-95ba392c4228 monverifyservicemanagement
0c677e82-5bfc-4e38-9155-e416249e3ee6 ms-catalog-postpyretic-pomiculture
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
However13 the13 translate13 plugin13 is13 not13 included13 with13 the13 Logstash13 installaon13 In13 the13 next13 secon13 I13 will13 13 describe13 how13 to13 add13 new13 plugins13 to13 the13 Logstash13 instance13
Add13 custom13 plugins13 to13 Logstash13 As13 an13 open13 source13 tool13 Logstash13 encourages13 users13 to13 develop13 content13 for13 all13 users13 to13 share13 Some13 of13 the13 plugins13 that13 were13 created13 by13 our13 community13 include13 a13 translaon13 plugin13 an13 output13 Slack13 plugin13 and13 an13 output13 syslog13 plugin13 Letrsquos13 look13 at13 how13 to13 add13 these13 plugins13 to13 your13 environment13
Filter13 translate13 plugin13 To13 add13 the13 translate13 plugin13
1 Download13 the13 plugin13
2 Copy13 the13 plugin13 file13 logstash-shy‐filter-shy‐translate-shy‐214gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-filter-translate 214
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Output13 Slack13 plugin13 To13 add13 the13 Slack13 plugin13
Page of 39 71 Page of 39 71
1 Download13 the13 plugin13 13
2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐slack-shy‐014gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-output-slack 014
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Output13 syslog13 plugin13 To13 add13 the13 syslog13 plugin13
1 Download13 the13 plugin13 13
2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐syslog-shy‐214gem13 to13 the13 ELK13 Virtual13 Server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-output-syslog 214
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Integrate13 Logstash13 with13 Slack13 To13 send13 generated13 alerts13 to13 Slack13 complete13 these13 steps13
1 Create13 a13 Slack13 integraon13 13
2 Use13 the13 Slack13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 Slack13 integraon13
113 Login13 to13 Slack13 and13 go13 to13 the13 Apps13 page13 13
Page of 40 71 Page of 40 71
13
313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13
13
A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13
13
Page of 42 71 Page of 42 71
The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13
13
The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13
13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
slack
url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5
channel =gt cloud-operations-mon
username =gt nick_cawood
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13
Page of 43 71 Page of 43 71
Page of 44 71 Page of 44 71
In13 Slack13 the13 alerts13 look13 like13 this13
13
Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13
1 Create13 a13 PagerDuty13 integraon13 13
2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13
1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13
13
2 Click13 Add13 New13 Service13
13
313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13
Page of 45 71 Page of 45 71
13
413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13
513 Click13 Add13 Service13
13
The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13
Page of 46 71 Page of 46 71
13
The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt be7eb1ea9c224018923e818d3e891d0b
incident_key =gt logstashhosttype
Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13
Page of 47 71 Page of 47 71
13
Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13
1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13
2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13
3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13
4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13
13 13 http
url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo
http_method =gt post
format =gt json
For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13
13 13 syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
Page of 48 71 Page of 48 71
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13
13
The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13
if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2
This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13
Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13
Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13
Logstash13 severity Shown13 in13 Omnibus13 as
alert Indeterminate
crical Warning
debug 7
emergency Clear
error Minor
informaonal 6
noce Crical
warning Major
Page of 49 71 Page of 49 71
13
Page of 50 71 Page of 50 71
Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13
input
tcp
port =gt 5000
type =gt syslog
filter
mutate
add_field =gt format =gt cf
grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
translate
field =gt data1
destination =gt data1
override =gt true
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
=== Check Alarm Criteria ===
Standard Key Words
if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
=== Alarm Exclusions ===
if [message] =~ (i)sometexthere
mutate remove_tag =gt servicename_alarm
Throttle alarms so we do not hit PD rate limits
Page of 52 71 Page of 52 71
if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 5
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle
after_count =gt 5
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 5
period =gt 1200
key =gt servicename_alarm_warning
add_tag =gt servicename_throttled
Page of 53 71 Page of 53 71
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
output
elasticsearch
hosts =gt localhost
stdout codec =gt rubydebug
if servicename_alarm_err in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
Page of 54 71 Page of 54 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 55 71 Page of 55 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
Page of 56 71 Page of 56 71
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert err message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_error in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
Page of 57 71 Page of 57 71
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 58 71 Page of 58 71
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 59 71 Page of 59 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert error message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
Page of 60 71 Page of 60 71
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
Page of 61 71 Page of 61 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 62 71 Page of 62 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt x
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert fatal message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
Page of 63 71 Page of 63 71
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
Page of 64 71 Page of 64 71
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 65 71 Page of 65 71
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt heavy_exclamation_mark
Page of 66 71 Page of 66 71
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert WARNING message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 67 71 Page of 67 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
Page of 68 71 Page of 68 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 69 71 Page of 69 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
Page of 70 71 Page of 70 71
Page of 71 71 Page of 71 71
Aler0ng13 with13 ELK13 ELK13 examines13 the13 log13 records13 in13 process13 to13 send13 alerts13 to13 third-shy‐party13 alerng13 event13 management13 and13 social13 media13 applicaons13 As13 shown13 previously13 in13 the13 ELK13 architecture13 the13 ELK13 strategy13 for13 alerng13 is13 to13 send13 alerts13 from13 the13 Logstash13 component13 directly13 Therefore13 log13 records13 and13 alerts13 are13 sent13 to13 both13 Elascsearch13 and13 third-shy‐party13 tools13 at13 the13 same13 instant13 using13 the13 output13 secon13 of13 the13 Logstash13 configuraon13 Any13 alerts13 are13 generated13 during13 the13 filter13 secon13 processing13 of13 the13 Logstash13 configuraon13
Create13 alerts13 in13 Logstash13 The13 Logstash13 filter13 secon13 is13 used13 to13 trigger13 alerts13 based13 on13 the13 log13 records13 being13 processed13 For13 example13 log13 records13 can13 be13 checked13 for13 errors13 warnings13 crashed13 or13 stopped13 messages13 and13 generate13 an13 alert13
Here13 is13 an13 example13 Logstash13 filter13 code13
13 if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
The13 specific13 tags13 are13 set13 against13 the13 log13 records13 to13 trigger13 the13 alert13 later13 in13 the13 configuraon13 in13 the13 output13 secon13
Control13 the13 number13 of13 generated13 alerts13 using13 the13 Logstash13 throcle13 plugin13 so13 that13 either13 a13 lot13 of13 instances13 in13 a13 fixed13 me13 period13 are13 required13 before13 an13 alert13 is13 sent13 or13 a13 specific13 number13 of13 instances13 triggers13 the13 alert13
Page of 36 71 Page of 36 71
Edit13 the13 plugin13 parameters13 to13 meet13 system13 requirements13
13 13 13 13 13 if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 10
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle13
after_count =gt 20
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 30
period =gt 1200
key =gt servicename_alarm_warning
Page of 37 71 Page of 37 71
add_tag =gt servicename_throttled
13 13 13 13 13 13 13 13 13
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 40
period =gt 1500
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
Enrich13 alerts13 with13 message13 detail13 Some13 of13 the13 third-shy‐party13 integraons13 benefit13 from13 having13 components13 of13 the13 raw13 log13 record13 sent13 in13 addion13 to13 the13 raw13 record13 or13 fields13 in13 the13 record13 edited13 transformed13 or13 translated13
Grok13 Logstash13 allows13 the13 log13 record13 to13 be13 broken13 up13 into13 logical13 secons13 and13 each13 secon13 stored13 in13 a13 new13 parameter13 using13 the13 grok13 parameter13 in13 the13 filter13 secon13 Here13 is13 an13 example13 grok13
13 grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
To13 build13 the13 grok13 statement13 Logstash13 offers13 an13 online13 tool13 at13 hcpsgrokdebugherokuappcom13
Translate13 If13 specific13 fields13 in13 the13 log13 record13 are13 coded13 (for13 example13 the13 Bluemix13 app13 name13 is13 oaen13 not13 sent13 in13 the13 syslog13 messages13 instead13 the13 GUID13 is13 sent13 which13 is13 not13 useful)13 then13 the13 filter13 secon13 translate13 plugin13 can13 be13 used13 to13 add13 new13 fields13 by13 holding13 the13 decoded13 versions13 of13 those13 values13
13 translate
field =gt data1
destination =gt data1
override =gt true
Page of 38 71 Page of 38 71
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
dc50f6e4-c052-436a-bf27-95ba392c4228 monverifyservicemanagement
0c677e82-5bfc-4e38-9155-e416249e3ee6 ms-catalog-postpyretic-pomiculture
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
However13 the13 translate13 plugin13 is13 not13 included13 with13 the13 Logstash13 installaon13 In13 the13 next13 secon13 I13 will13 13 describe13 how13 to13 add13 new13 plugins13 to13 the13 Logstash13 instance13
Add13 custom13 plugins13 to13 Logstash13 As13 an13 open13 source13 tool13 Logstash13 encourages13 users13 to13 develop13 content13 for13 all13 users13 to13 share13 Some13 of13 the13 plugins13 that13 were13 created13 by13 our13 community13 include13 a13 translaon13 plugin13 an13 output13 Slack13 plugin13 and13 an13 output13 syslog13 plugin13 Letrsquos13 look13 at13 how13 to13 add13 these13 plugins13 to13 your13 environment13
Filter13 translate13 plugin13 To13 add13 the13 translate13 plugin13
1 Download13 the13 plugin13
2 Copy13 the13 plugin13 file13 logstash-shy‐filter-shy‐translate-shy‐214gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-filter-translate 214
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Output13 Slack13 plugin13 To13 add13 the13 Slack13 plugin13
Page of 39 71 Page of 39 71
1 Download13 the13 plugin13 13
2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐slack-shy‐014gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-output-slack 014
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Output13 syslog13 plugin13 To13 add13 the13 syslog13 plugin13
1 Download13 the13 plugin13 13
2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐syslog-shy‐214gem13 to13 the13 ELK13 Virtual13 Server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-output-syslog 214
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Integrate13 Logstash13 with13 Slack13 To13 send13 generated13 alerts13 to13 Slack13 complete13 these13 steps13
1 Create13 a13 Slack13 integraon13 13
2 Use13 the13 Slack13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 Slack13 integraon13
113 Login13 to13 Slack13 and13 go13 to13 the13 Apps13 page13 13
Page of 40 71 Page of 40 71
13
313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13
13
A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13
13
Page of 42 71 Page of 42 71
The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13
13
The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13
13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
slack
url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5
channel =gt cloud-operations-mon
username =gt nick_cawood
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13
Page of 43 71 Page of 43 71
Page of 44 71 Page of 44 71
In13 Slack13 the13 alerts13 look13 like13 this13
13
Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13
1 Create13 a13 PagerDuty13 integraon13 13
2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13
1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13
13
2 Click13 Add13 New13 Service13
13
313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13
Page of 45 71 Page of 45 71
13
413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13
513 Click13 Add13 Service13
13
The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13
Page of 46 71 Page of 46 71
13
The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt be7eb1ea9c224018923e818d3e891d0b
incident_key =gt logstashhosttype
Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13
Page of 47 71 Page of 47 71
13
Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13
1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13
2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13
3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13
4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13
13 13 http
url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo
http_method =gt post
format =gt json
For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13
13 13 syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
Page of 48 71 Page of 48 71
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13
13
The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13
if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2
This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13
Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13
Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13
Logstash13 severity Shown13 in13 Omnibus13 as
alert Indeterminate
crical Warning
debug 7
emergency Clear
error Minor
informaonal 6
noce Crical
warning Major
Page of 49 71 Page of 49 71
13
Page of 50 71 Page of 50 71
Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13
input
tcp
port =gt 5000
type =gt syslog
filter
mutate
add_field =gt format =gt cf
grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
translate
field =gt data1
destination =gt data1
override =gt true
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
=== Check Alarm Criteria ===
Standard Key Words
if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
=== Alarm Exclusions ===
if [message] =~ (i)sometexthere
mutate remove_tag =gt servicename_alarm
Throttle alarms so we do not hit PD rate limits
Page of 52 71 Page of 52 71
if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 5
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle
after_count =gt 5
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 5
period =gt 1200
key =gt servicename_alarm_warning
add_tag =gt servicename_throttled
Page of 53 71 Page of 53 71
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
output
elasticsearch
hosts =gt localhost
stdout codec =gt rubydebug
if servicename_alarm_err in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
Page of 54 71 Page of 54 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 55 71 Page of 55 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
Page of 56 71 Page of 56 71
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert err message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_error in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
Page of 57 71 Page of 57 71
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 58 71 Page of 58 71
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 59 71 Page of 59 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert error message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
Page of 60 71 Page of 60 71
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
Page of 61 71 Page of 61 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 62 71 Page of 62 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt x
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert fatal message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
Page of 63 71 Page of 63 71
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
Page of 64 71 Page of 64 71
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 65 71 Page of 65 71
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt heavy_exclamation_mark
Page of 66 71 Page of 66 71
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert WARNING message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 67 71 Page of 67 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
Page of 68 71 Page of 68 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 69 71 Page of 69 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
Page of 70 71 Page of 70 71
Page of 71 71 Page of 71 71
Edit13 the13 plugin13 parameters13 to13 meet13 system13 requirements13
13 13 13 13 13 if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 10
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle13
after_count =gt 20
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 30
period =gt 1200
key =gt servicename_alarm_warning
Page of 37 71 Page of 37 71
add_tag =gt servicename_throttled
13 13 13 13 13 13 13 13 13
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 40
period =gt 1500
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
Enrich13 alerts13 with13 message13 detail13 Some13 of13 the13 third-shy‐party13 integraons13 benefit13 from13 having13 components13 of13 the13 raw13 log13 record13 sent13 in13 addion13 to13 the13 raw13 record13 or13 fields13 in13 the13 record13 edited13 transformed13 or13 translated13
Grok13 Logstash13 allows13 the13 log13 record13 to13 be13 broken13 up13 into13 logical13 secons13 and13 each13 secon13 stored13 in13 a13 new13 parameter13 using13 the13 grok13 parameter13 in13 the13 filter13 secon13 Here13 is13 an13 example13 grok13
13 grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
To13 build13 the13 grok13 statement13 Logstash13 offers13 an13 online13 tool13 at13 hcpsgrokdebugherokuappcom13
Translate13 If13 specific13 fields13 in13 the13 log13 record13 are13 coded13 (for13 example13 the13 Bluemix13 app13 name13 is13 oaen13 not13 sent13 in13 the13 syslog13 messages13 instead13 the13 GUID13 is13 sent13 which13 is13 not13 useful)13 then13 the13 filter13 secon13 translate13 plugin13 can13 be13 used13 to13 add13 new13 fields13 by13 holding13 the13 decoded13 versions13 of13 those13 values13
13 translate
field =gt data1
destination =gt data1
override =gt true
Page of 38 71 Page of 38 71
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
dc50f6e4-c052-436a-bf27-95ba392c4228 monverifyservicemanagement
0c677e82-5bfc-4e38-9155-e416249e3ee6 ms-catalog-postpyretic-pomiculture
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
However13 the13 translate13 plugin13 is13 not13 included13 with13 the13 Logstash13 installaon13 In13 the13 next13 secon13 I13 will13 13 describe13 how13 to13 add13 new13 plugins13 to13 the13 Logstash13 instance13
Add13 custom13 plugins13 to13 Logstash13 As13 an13 open13 source13 tool13 Logstash13 encourages13 users13 to13 develop13 content13 for13 all13 users13 to13 share13 Some13 of13 the13 plugins13 that13 were13 created13 by13 our13 community13 include13 a13 translaon13 plugin13 an13 output13 Slack13 plugin13 and13 an13 output13 syslog13 plugin13 Letrsquos13 look13 at13 how13 to13 add13 these13 plugins13 to13 your13 environment13
Filter13 translate13 plugin13 To13 add13 the13 translate13 plugin13
1 Download13 the13 plugin13
2 Copy13 the13 plugin13 file13 logstash-shy‐filter-shy‐translate-shy‐214gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-filter-translate 214
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Output13 Slack13 plugin13 To13 add13 the13 Slack13 plugin13
Page of 39 71 Page of 39 71
1 Download13 the13 plugin13 13
2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐slack-shy‐014gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-output-slack 014
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Output13 syslog13 plugin13 To13 add13 the13 syslog13 plugin13
1 Download13 the13 plugin13 13
2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐syslog-shy‐214gem13 to13 the13 ELK13 Virtual13 Server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-output-syslog 214
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Integrate13 Logstash13 with13 Slack13 To13 send13 generated13 alerts13 to13 Slack13 complete13 these13 steps13
1 Create13 a13 Slack13 integraon13 13
2 Use13 the13 Slack13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 Slack13 integraon13
113 Login13 to13 Slack13 and13 go13 to13 the13 Apps13 page13 13
Page of 40 71 Page of 40 71
13
313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13
13
A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13
13
Page of 42 71 Page of 42 71
The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13
13
The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13
13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
slack
url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5
channel =gt cloud-operations-mon
username =gt nick_cawood
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13
Page of 43 71 Page of 43 71
Page of 44 71 Page of 44 71
In13 Slack13 the13 alerts13 look13 like13 this13
13
Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13
1 Create13 a13 PagerDuty13 integraon13 13
2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13
1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13
13
2 Click13 Add13 New13 Service13
13
313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13
Page of 45 71 Page of 45 71
13
413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13
513 Click13 Add13 Service13
13
The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13
Page of 46 71 Page of 46 71
13
The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt be7eb1ea9c224018923e818d3e891d0b
incident_key =gt logstashhosttype
Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13
Page of 47 71 Page of 47 71
13
Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13
1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13
2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13
3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13
4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13
13 13 http
url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo
http_method =gt post
format =gt json
For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13
13 13 syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
Page of 48 71 Page of 48 71
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13
13
The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13
if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2
This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13
Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13
Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13
Logstash13 severity Shown13 in13 Omnibus13 as
alert Indeterminate
crical Warning
debug 7
emergency Clear
error Minor
informaonal 6
noce Crical
warning Major
Page of 49 71 Page of 49 71
13
Page of 50 71 Page of 50 71
Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13
input
tcp
port =gt 5000
type =gt syslog
filter
mutate
add_field =gt format =gt cf
grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
translate
field =gt data1
destination =gt data1
override =gt true
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
=== Check Alarm Criteria ===
Standard Key Words
if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
=== Alarm Exclusions ===
if [message] =~ (i)sometexthere
mutate remove_tag =gt servicename_alarm
Throttle alarms so we do not hit PD rate limits
Page of 52 71 Page of 52 71
if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 5
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle
after_count =gt 5
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 5
period =gt 1200
key =gt servicename_alarm_warning
add_tag =gt servicename_throttled
Page of 53 71 Page of 53 71
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
output
elasticsearch
hosts =gt localhost
stdout codec =gt rubydebug
if servicename_alarm_err in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
Page of 54 71 Page of 54 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 55 71 Page of 55 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
Page of 56 71 Page of 56 71
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert err message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_error in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
Page of 57 71 Page of 57 71
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 58 71 Page of 58 71
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 59 71 Page of 59 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert error message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
Page of 60 71 Page of 60 71
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
Page of 61 71 Page of 61 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 62 71 Page of 62 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt x
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert fatal message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
Page of 63 71 Page of 63 71
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
Page of 64 71 Page of 64 71
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 65 71 Page of 65 71
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt heavy_exclamation_mark
Page of 66 71 Page of 66 71
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert WARNING message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 67 71 Page of 67 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
Page of 68 71 Page of 68 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 69 71 Page of 69 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
Page of 70 71 Page of 70 71
Page of 71 71 Page of 71 71
add_tag =gt servicename_throttled
13 13 13 13 13 13 13 13 13
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 40
period =gt 1500
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
Enrich13 alerts13 with13 message13 detail13 Some13 of13 the13 third-shy‐party13 integraons13 benefit13 from13 having13 components13 of13 the13 raw13 log13 record13 sent13 in13 addion13 to13 the13 raw13 record13 or13 fields13 in13 the13 record13 edited13 transformed13 or13 translated13
Grok13 Logstash13 allows13 the13 log13 record13 to13 be13 broken13 up13 into13 logical13 secons13 and13 each13 secon13 stored13 in13 a13 new13 parameter13 using13 the13 grok13 parameter13 in13 the13 filter13 secon13 Here13 is13 an13 example13 grok13
13 grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
To13 build13 the13 grok13 statement13 Logstash13 offers13 an13 online13 tool13 at13 hcpsgrokdebugherokuappcom13
Translate13 If13 specific13 fields13 in13 the13 log13 record13 are13 coded13 (for13 example13 the13 Bluemix13 app13 name13 is13 oaen13 not13 sent13 in13 the13 syslog13 messages13 instead13 the13 GUID13 is13 sent13 which13 is13 not13 useful)13 then13 the13 filter13 secon13 translate13 plugin13 can13 be13 used13 to13 add13 new13 fields13 by13 holding13 the13 decoded13 versions13 of13 those13 values13
13 translate
field =gt data1
destination =gt data1
override =gt true
Page of 38 71 Page of 38 71
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
dc50f6e4-c052-436a-bf27-95ba392c4228 monverifyservicemanagement
0c677e82-5bfc-4e38-9155-e416249e3ee6 ms-catalog-postpyretic-pomiculture
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
However13 the13 translate13 plugin13 is13 not13 included13 with13 the13 Logstash13 installaon13 In13 the13 next13 secon13 I13 will13 13 describe13 how13 to13 add13 new13 plugins13 to13 the13 Logstash13 instance13
Add13 custom13 plugins13 to13 Logstash13 As13 an13 open13 source13 tool13 Logstash13 encourages13 users13 to13 develop13 content13 for13 all13 users13 to13 share13 Some13 of13 the13 plugins13 that13 were13 created13 by13 our13 community13 include13 a13 translaon13 plugin13 an13 output13 Slack13 plugin13 and13 an13 output13 syslog13 plugin13 Letrsquos13 look13 at13 how13 to13 add13 these13 plugins13 to13 your13 environment13
Filter13 translate13 plugin13 To13 add13 the13 translate13 plugin13
1 Download13 the13 plugin13
2 Copy13 the13 plugin13 file13 logstash-shy‐filter-shy‐translate-shy‐214gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-filter-translate 214
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Output13 Slack13 plugin13 To13 add13 the13 Slack13 plugin13
Page of 39 71 Page of 39 71
1 Download13 the13 plugin13 13
2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐slack-shy‐014gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-output-slack 014
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Output13 syslog13 plugin13 To13 add13 the13 syslog13 plugin13
1 Download13 the13 plugin13 13
2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐syslog-shy‐214gem13 to13 the13 ELK13 Virtual13 Server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-output-syslog 214
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Integrate13 Logstash13 with13 Slack13 To13 send13 generated13 alerts13 to13 Slack13 complete13 these13 steps13
1 Create13 a13 Slack13 integraon13 13
2 Use13 the13 Slack13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 Slack13 integraon13
113 Login13 to13 Slack13 and13 go13 to13 the13 Apps13 page13 13
Page of 40 71 Page of 40 71
13
313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13
13
A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13
13
Page of 42 71 Page of 42 71
The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13
13
The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13
13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
slack
url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5
channel =gt cloud-operations-mon
username =gt nick_cawood
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13
Page of 43 71 Page of 43 71
Page of 44 71 Page of 44 71
In13 Slack13 the13 alerts13 look13 like13 this13
13
Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13
1 Create13 a13 PagerDuty13 integraon13 13
2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13
1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13
13
2 Click13 Add13 New13 Service13
13
313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13
Page of 45 71 Page of 45 71
13
413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13
513 Click13 Add13 Service13
13
The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13
Page of 46 71 Page of 46 71
13
The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt be7eb1ea9c224018923e818d3e891d0b
incident_key =gt logstashhosttype
Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13
Page of 47 71 Page of 47 71
13
Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13
1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13
2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13
3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13
4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13
13 13 http
url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo
http_method =gt post
format =gt json
For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13
13 13 syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
Page of 48 71 Page of 48 71
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13
13
The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13
if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2
This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13
Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13
Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13
Logstash13 severity Shown13 in13 Omnibus13 as
alert Indeterminate
crical Warning
debug 7
emergency Clear
error Minor
informaonal 6
noce Crical
warning Major
Page of 49 71 Page of 49 71
13
Page of 50 71 Page of 50 71
Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13
input
tcp
port =gt 5000
type =gt syslog
filter
mutate
add_field =gt format =gt cf
grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
translate
field =gt data1
destination =gt data1
override =gt true
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
=== Check Alarm Criteria ===
Standard Key Words
if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
=== Alarm Exclusions ===
if [message] =~ (i)sometexthere
mutate remove_tag =gt servicename_alarm
Throttle alarms so we do not hit PD rate limits
Page of 52 71 Page of 52 71
if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 5
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle
after_count =gt 5
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 5
period =gt 1200
key =gt servicename_alarm_warning
add_tag =gt servicename_throttled
Page of 53 71 Page of 53 71
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
output
elasticsearch
hosts =gt localhost
stdout codec =gt rubydebug
if servicename_alarm_err in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
Page of 54 71 Page of 54 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 55 71 Page of 55 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
Page of 56 71 Page of 56 71
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert err message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_error in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
Page of 57 71 Page of 57 71
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 58 71 Page of 58 71
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 59 71 Page of 59 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert error message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
Page of 60 71 Page of 60 71
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
Page of 61 71 Page of 61 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 62 71 Page of 62 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt x
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert fatal message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
Page of 63 71 Page of 63 71
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
Page of 64 71 Page of 64 71
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 65 71 Page of 65 71
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt heavy_exclamation_mark
Page of 66 71 Page of 66 71
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert WARNING message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 67 71 Page of 67 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
Page of 68 71 Page of 68 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 69 71 Page of 69 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
Page of 70 71 Page of 70 71
Page of 71 71 Page of 71 71
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
dc50f6e4-c052-436a-bf27-95ba392c4228 monverifyservicemanagement
0c677e82-5bfc-4e38-9155-e416249e3ee6 ms-catalog-postpyretic-pomiculture
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
However13 the13 translate13 plugin13 is13 not13 included13 with13 the13 Logstash13 installaon13 In13 the13 next13 secon13 I13 will13 13 describe13 how13 to13 add13 new13 plugins13 to13 the13 Logstash13 instance13
Add13 custom13 plugins13 to13 Logstash13 As13 an13 open13 source13 tool13 Logstash13 encourages13 users13 to13 develop13 content13 for13 all13 users13 to13 share13 Some13 of13 the13 plugins13 that13 were13 created13 by13 our13 community13 include13 a13 translaon13 plugin13 an13 output13 Slack13 plugin13 and13 an13 output13 syslog13 plugin13 Letrsquos13 look13 at13 how13 to13 add13 these13 plugins13 to13 your13 environment13
Filter13 translate13 plugin13 To13 add13 the13 translate13 plugin13
1 Download13 the13 plugin13
2 Copy13 the13 plugin13 file13 logstash-shy‐filter-shy‐translate-shy‐214gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-filter-translate 214
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Output13 Slack13 plugin13 To13 add13 the13 Slack13 plugin13
Page of 39 71 Page of 39 71
1 Download13 the13 plugin13 13
2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐slack-shy‐014gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-output-slack 014
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Output13 syslog13 plugin13 To13 add13 the13 syslog13 plugin13
1 Download13 the13 plugin13 13
2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐syslog-shy‐214gem13 to13 the13 ELK13 Virtual13 Server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-output-syslog 214
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Integrate13 Logstash13 with13 Slack13 To13 send13 generated13 alerts13 to13 Slack13 complete13 these13 steps13
1 Create13 a13 Slack13 integraon13 13
2 Use13 the13 Slack13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 Slack13 integraon13
113 Login13 to13 Slack13 and13 go13 to13 the13 Apps13 page13 13
Page of 40 71 Page of 40 71
13
313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13
13
A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13
13
Page of 42 71 Page of 42 71
The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13
13
The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13
13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
slack
url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5
channel =gt cloud-operations-mon
username =gt nick_cawood
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13
Page of 43 71 Page of 43 71
Page of 44 71 Page of 44 71
In13 Slack13 the13 alerts13 look13 like13 this13
13
Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13
1 Create13 a13 PagerDuty13 integraon13 13
2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13
1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13
13
2 Click13 Add13 New13 Service13
13
313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13
Page of 45 71 Page of 45 71
13
413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13
513 Click13 Add13 Service13
13
The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13
Page of 46 71 Page of 46 71
13
The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt be7eb1ea9c224018923e818d3e891d0b
incident_key =gt logstashhosttype
Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13
Page of 47 71 Page of 47 71
13
Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13
1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13
2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13
3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13
4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13
13 13 http
url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo
http_method =gt post
format =gt json
For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13
13 13 syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
Page of 48 71 Page of 48 71
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13
13
The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13
if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2
This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13
Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13
Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13
Logstash13 severity Shown13 in13 Omnibus13 as
alert Indeterminate
crical Warning
debug 7
emergency Clear
error Minor
informaonal 6
noce Crical
warning Major
Page of 49 71 Page of 49 71
13
Page of 50 71 Page of 50 71
Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13
input
tcp
port =gt 5000
type =gt syslog
filter
mutate
add_field =gt format =gt cf
grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
translate
field =gt data1
destination =gt data1
override =gt true
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
=== Check Alarm Criteria ===
Standard Key Words
if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
=== Alarm Exclusions ===
if [message] =~ (i)sometexthere
mutate remove_tag =gt servicename_alarm
Throttle alarms so we do not hit PD rate limits
Page of 52 71 Page of 52 71
if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 5
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle
after_count =gt 5
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 5
period =gt 1200
key =gt servicename_alarm_warning
add_tag =gt servicename_throttled
Page of 53 71 Page of 53 71
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
output
elasticsearch
hosts =gt localhost
stdout codec =gt rubydebug
if servicename_alarm_err in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
Page of 54 71 Page of 54 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 55 71 Page of 55 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
Page of 56 71 Page of 56 71
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert err message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_error in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
Page of 57 71 Page of 57 71
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 58 71 Page of 58 71
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 59 71 Page of 59 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert error message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
Page of 60 71 Page of 60 71
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
Page of 61 71 Page of 61 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 62 71 Page of 62 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt x
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert fatal message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
Page of 63 71 Page of 63 71
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
Page of 64 71 Page of 64 71
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 65 71 Page of 65 71
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt heavy_exclamation_mark
Page of 66 71 Page of 66 71
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert WARNING message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 67 71 Page of 67 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
Page of 68 71 Page of 68 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 69 71 Page of 69 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
Page of 70 71 Page of 70 71
Page of 71 71 Page of 71 71
1 Download13 the13 plugin13 13
2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐slack-shy‐014gem13 to13 the13 ELK13 virtual13 server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-output-slack 014
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Output13 syslog13 plugin13 To13 add13 the13 syslog13 plugin13
1 Download13 the13 plugin13 13
2 Copy13 the13 plugin13 file13 logstash-shy‐output-shy‐syslog-shy‐214gem13 to13 the13 ELK13 Virtual13 Server13 (see13 previous13 secon13 in13 this13 document)13
3 As13 a13 root13 user13 move13 the13 file13 to13 homelogstashlogstash-shy‐233plugins13 then13 edit13 the13 file13 homelogstashlogstash-shy‐233Gemfile13 to13 add13 the13 line13
gem logstash-output-syslog 214
4 Run13 the13 command13
[rootelkforcase logstash-233] homelogstashlogstash-233binlogstash-plugin install --no-verify
The13 plugin13 installs13 and13 is13 ready13 to13 use13 in13 a13 Logstash13 filter13
Integrate13 Logstash13 with13 Slack13 To13 send13 generated13 alerts13 to13 Slack13 complete13 these13 steps13
1 Create13 a13 Slack13 integraon13 13
2 Use13 the13 Slack13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 Slack13 integraon13
113 Login13 to13 Slack13 and13 go13 to13 the13 Apps13 page13 13
Page of 40 71 Page of 40 71
13
313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13
13
A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13
13
Page of 42 71 Page of 42 71
The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13
13
The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13
13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
slack
url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5
channel =gt cloud-operations-mon
username =gt nick_cawood
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13
Page of 43 71 Page of 43 71
Page of 44 71 Page of 44 71
In13 Slack13 the13 alerts13 look13 like13 this13
13
Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13
1 Create13 a13 PagerDuty13 integraon13 13
2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13
1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13
13
2 Click13 Add13 New13 Service13
13
313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13
Page of 45 71 Page of 45 71
13
413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13
513 Click13 Add13 Service13
13
The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13
Page of 46 71 Page of 46 71
13
The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt be7eb1ea9c224018923e818d3e891d0b
incident_key =gt logstashhosttype
Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13
Page of 47 71 Page of 47 71
13
Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13
1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13
2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13
3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13
4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13
13 13 http
url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo
http_method =gt post
format =gt json
For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13
13 13 syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
Page of 48 71 Page of 48 71
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13
13
The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13
if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2
This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13
Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13
Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13
Logstash13 severity Shown13 in13 Omnibus13 as
alert Indeterminate
crical Warning
debug 7
emergency Clear
error Minor
informaonal 6
noce Crical
warning Major
Page of 49 71 Page of 49 71
13
Page of 50 71 Page of 50 71
Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13
input
tcp
port =gt 5000
type =gt syslog
filter
mutate
add_field =gt format =gt cf
grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
translate
field =gt data1
destination =gt data1
override =gt true
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
=== Check Alarm Criteria ===
Standard Key Words
if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
=== Alarm Exclusions ===
if [message] =~ (i)sometexthere
mutate remove_tag =gt servicename_alarm
Throttle alarms so we do not hit PD rate limits
Page of 52 71 Page of 52 71
if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 5
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle
after_count =gt 5
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 5
period =gt 1200
key =gt servicename_alarm_warning
add_tag =gt servicename_throttled
Page of 53 71 Page of 53 71
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
output
elasticsearch
hosts =gt localhost
stdout codec =gt rubydebug
if servicename_alarm_err in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
Page of 54 71 Page of 54 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 55 71 Page of 55 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
Page of 56 71 Page of 56 71
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert err message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_error in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
Page of 57 71 Page of 57 71
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 58 71 Page of 58 71
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 59 71 Page of 59 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert error message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
Page of 60 71 Page of 60 71
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
Page of 61 71 Page of 61 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 62 71 Page of 62 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt x
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert fatal message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
Page of 63 71 Page of 63 71
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
Page of 64 71 Page of 64 71
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 65 71 Page of 65 71
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt heavy_exclamation_mark
Page of 66 71 Page of 66 71
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert WARNING message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 67 71 Page of 67 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
Page of 68 71 Page of 68 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 69 71 Page of 69 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
Page of 70 71 Page of 70 71
Page of 71 71 Page of 71 71
13
313 To13 add13 a13 new13 WebHook13 click13 Add13 Configura0on13 and13 select13 a13 Slack13 channel13 to13 integrate13 with13
13
A13 WebHook13 is13 created13 with13 the13 Webhook13 URL13 to13 use13 to13 integrate13 Logstash13 with13 Slack13
13
Page of 42 71 Page of 42 71
The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13
13
The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13
13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
slack
url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5
channel =gt cloud-operations-mon
username =gt nick_cawood
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13
Page of 43 71 Page of 43 71
Page of 44 71 Page of 44 71
In13 Slack13 the13 alerts13 look13 like13 this13
13
Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13
1 Create13 a13 PagerDuty13 integraon13 13
2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13
1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13
13
2 Click13 Add13 New13 Service13
13
313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13
Page of 45 71 Page of 45 71
13
413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13
513 Click13 Add13 Service13
13
The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13
Page of 46 71 Page of 46 71
13
The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt be7eb1ea9c224018923e818d3e891d0b
incident_key =gt logstashhosttype
Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13
Page of 47 71 Page of 47 71
13
Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13
1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13
2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13
3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13
4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13
13 13 http
url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo
http_method =gt post
format =gt json
For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13
13 13 syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
Page of 48 71 Page of 48 71
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13
13
The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13
if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2
This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13
Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13
Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13
Logstash13 severity Shown13 in13 Omnibus13 as
alert Indeterminate
crical Warning
debug 7
emergency Clear
error Minor
informaonal 6
noce Crical
warning Major
Page of 49 71 Page of 49 71
13
Page of 50 71 Page of 50 71
Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13
input
tcp
port =gt 5000
type =gt syslog
filter
mutate
add_field =gt format =gt cf
grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
translate
field =gt data1
destination =gt data1
override =gt true
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
=== Check Alarm Criteria ===
Standard Key Words
if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
=== Alarm Exclusions ===
if [message] =~ (i)sometexthere
mutate remove_tag =gt servicename_alarm
Throttle alarms so we do not hit PD rate limits
Page of 52 71 Page of 52 71
if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 5
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle
after_count =gt 5
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 5
period =gt 1200
key =gt servicename_alarm_warning
add_tag =gt servicename_throttled
Page of 53 71 Page of 53 71
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
output
elasticsearch
hosts =gt localhost
stdout codec =gt rubydebug
if servicename_alarm_err in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
Page of 54 71 Page of 54 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 55 71 Page of 55 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
Page of 56 71 Page of 56 71
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert err message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_error in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
Page of 57 71 Page of 57 71
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 58 71 Page of 58 71
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 59 71 Page of 59 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert error message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
Page of 60 71 Page of 60 71
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
Page of 61 71 Page of 61 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 62 71 Page of 62 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt x
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert fatal message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
Page of 63 71 Page of 63 71
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
Page of 64 71 Page of 64 71
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 65 71 Page of 65 71
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt heavy_exclamation_mark
Page of 66 71 Page of 66 71
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert WARNING message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 67 71 Page of 67 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
Page of 68 71 Page of 68 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 69 71 Page of 69 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
Page of 70 71 Page of 70 71
Page of 71 71 Page of 71 71
The13 new13 WebHook13 is13 listed13 in13 the13 App13 directory13
13
The13 generated13 URL13 can13 be13 used13 in13 the13 Logstash13 output13 Slack13 plugin13 specifying13 the13 Slack13 channel13 the13 Slack13 user13 and13 a13 custom13 message13 (using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output)13
13 if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
slack
url =gt httpshooksslackcomservicesT0NKYQPGBB1QTQ3H9PR9ZRWmQUAiJ1H5s3Ur5AgSs5
channel =gt cloud-operations-mon
username =gt nick_cawood
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
For13 icon13 details13 check13 the13 Internet13 for13 usable13 values13 for13 example13 the13 Emoji13 Cheat13 Sheet13 provides13 a13 list13 of13 emocons13
Page of 43 71 Page of 43 71
Page of 44 71 Page of 44 71
In13 Slack13 the13 alerts13 look13 like13 this13
13
Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13
1 Create13 a13 PagerDuty13 integraon13 13
2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13
1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13
13
2 Click13 Add13 New13 Service13
13
313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13
Page of 45 71 Page of 45 71
13
413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13
513 Click13 Add13 Service13
13
The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13
Page of 46 71 Page of 46 71
13
The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt be7eb1ea9c224018923e818d3e891d0b
incident_key =gt logstashhosttype
Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13
Page of 47 71 Page of 47 71
13
Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13
1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13
2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13
3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13
4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13
13 13 http
url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo
http_method =gt post
format =gt json
For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13
13 13 syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
Page of 48 71 Page of 48 71
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13
13
The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13
if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2
This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13
Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13
Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13
Logstash13 severity Shown13 in13 Omnibus13 as
alert Indeterminate
crical Warning
debug 7
emergency Clear
error Minor
informaonal 6
noce Crical
warning Major
Page of 49 71 Page of 49 71
13
Page of 50 71 Page of 50 71
Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13
input
tcp
port =gt 5000
type =gt syslog
filter
mutate
add_field =gt format =gt cf
grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
translate
field =gt data1
destination =gt data1
override =gt true
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
=== Check Alarm Criteria ===
Standard Key Words
if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
=== Alarm Exclusions ===
if [message] =~ (i)sometexthere
mutate remove_tag =gt servicename_alarm
Throttle alarms so we do not hit PD rate limits
Page of 52 71 Page of 52 71
if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 5
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle
after_count =gt 5
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 5
period =gt 1200
key =gt servicename_alarm_warning
add_tag =gt servicename_throttled
Page of 53 71 Page of 53 71
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
output
elasticsearch
hosts =gt localhost
stdout codec =gt rubydebug
if servicename_alarm_err in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
Page of 54 71 Page of 54 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 55 71 Page of 55 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
Page of 56 71 Page of 56 71
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert err message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_error in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
Page of 57 71 Page of 57 71
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 58 71 Page of 58 71
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 59 71 Page of 59 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert error message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
Page of 60 71 Page of 60 71
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
Page of 61 71 Page of 61 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 62 71 Page of 62 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt x
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert fatal message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
Page of 63 71 Page of 63 71
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
Page of 64 71 Page of 64 71
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 65 71 Page of 65 71
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt heavy_exclamation_mark
Page of 66 71 Page of 66 71
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert WARNING message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 67 71 Page of 67 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
Page of 68 71 Page of 68 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 69 71 Page of 69 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
Page of 70 71 Page of 70 71
Page of 71 71 Page of 71 71
Page of 44 71 Page of 44 71
In13 Slack13 the13 alerts13 look13 like13 this13
13
Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13
1 Create13 a13 PagerDuty13 integraon13 13
2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13
1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13
13
2 Click13 Add13 New13 Service13
13
313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13
Page of 45 71 Page of 45 71
13
413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13
513 Click13 Add13 Service13
13
The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13
Page of 46 71 Page of 46 71
13
The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt be7eb1ea9c224018923e818d3e891d0b
incident_key =gt logstashhosttype
Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13
Page of 47 71 Page of 47 71
13
Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13
1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13
2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13
3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13
4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13
13 13 http
url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo
http_method =gt post
format =gt json
For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13
13 13 syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
Page of 48 71 Page of 48 71
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13
13
The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13
if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2
This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13
Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13
Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13
Logstash13 severity Shown13 in13 Omnibus13 as
alert Indeterminate
crical Warning
debug 7
emergency Clear
error Minor
informaonal 6
noce Crical
warning Major
Page of 49 71 Page of 49 71
13
Page of 50 71 Page of 50 71
Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13
input
tcp
port =gt 5000
type =gt syslog
filter
mutate
add_field =gt format =gt cf
grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
translate
field =gt data1
destination =gt data1
override =gt true
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
=== Check Alarm Criteria ===
Standard Key Words
if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
=== Alarm Exclusions ===
if [message] =~ (i)sometexthere
mutate remove_tag =gt servicename_alarm
Throttle alarms so we do not hit PD rate limits
Page of 52 71 Page of 52 71
if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 5
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle
after_count =gt 5
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 5
period =gt 1200
key =gt servicename_alarm_warning
add_tag =gt servicename_throttled
Page of 53 71 Page of 53 71
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
output
elasticsearch
hosts =gt localhost
stdout codec =gt rubydebug
if servicename_alarm_err in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
Page of 54 71 Page of 54 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 55 71 Page of 55 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
Page of 56 71 Page of 56 71
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert err message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_error in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
Page of 57 71 Page of 57 71
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 58 71 Page of 58 71
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 59 71 Page of 59 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert error message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
Page of 60 71 Page of 60 71
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
Page of 61 71 Page of 61 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 62 71 Page of 62 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt x
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert fatal message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
Page of 63 71 Page of 63 71
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
Page of 64 71 Page of 64 71
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 65 71 Page of 65 71
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt heavy_exclamation_mark
Page of 66 71 Page of 66 71
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert WARNING message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 67 71 Page of 67 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
Page of 68 71 Page of 68 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 69 71 Page of 69 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
Page of 70 71 Page of 70 71
Page of 71 71 Page of 71 71
In13 Slack13 the13 alerts13 look13 like13 this13
13
Integrate13 Logstash13 with13 PagerDuty13 To13 send13 generated13 alerts13 to13 PagerDuty13 complete13 these13 steps13
1 Create13 a13 PagerDuty13 integraon13 13
2 Use13 the13 pagerduty13 plugin13 in13 the13 Logstash13 output13
To13 create13 the13 PagerDuty13 integraon13 follow13 these13 steps13
1 Log13 in13 to13 PagerDuty13 and13 select13 Configura0on13 agrave13 Services13
13
2 Click13 Add13 New13 Service13
13
313 Fill13 in13 a13 name13 descripon13 and13 select13 Logstash13 13 for13 the13 Integraon13 Type13
Page of 45 71 Page of 45 71
13
413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13
513 Click13 Add13 Service13
13
The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13
Page of 46 71 Page of 46 71
13
The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt be7eb1ea9c224018923e818d3e891d0b
incident_key =gt logstashhosttype
Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13
Page of 47 71 Page of 47 71
13
Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13
1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13
2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13
3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13
4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13
13 13 http
url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo
http_method =gt post
format =gt json
For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13
13 13 syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
Page of 48 71 Page of 48 71
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13
13
The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13
if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2
This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13
Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13
Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13
Logstash13 severity Shown13 in13 Omnibus13 as
alert Indeterminate
crical Warning
debug 7
emergency Clear
error Minor
informaonal 6
noce Crical
warning Major
Page of 49 71 Page of 49 71
13
Page of 50 71 Page of 50 71
Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13
input
tcp
port =gt 5000
type =gt syslog
filter
mutate
add_field =gt format =gt cf
grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
translate
field =gt data1
destination =gt data1
override =gt true
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
=== Check Alarm Criteria ===
Standard Key Words
if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
=== Alarm Exclusions ===
if [message] =~ (i)sometexthere
mutate remove_tag =gt servicename_alarm
Throttle alarms so we do not hit PD rate limits
Page of 52 71 Page of 52 71
if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 5
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle
after_count =gt 5
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 5
period =gt 1200
key =gt servicename_alarm_warning
add_tag =gt servicename_throttled
Page of 53 71 Page of 53 71
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
output
elasticsearch
hosts =gt localhost
stdout codec =gt rubydebug
if servicename_alarm_err in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
Page of 54 71 Page of 54 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 55 71 Page of 55 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
Page of 56 71 Page of 56 71
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert err message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_error in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
Page of 57 71 Page of 57 71
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 58 71 Page of 58 71
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 59 71 Page of 59 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert error message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
Page of 60 71 Page of 60 71
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
Page of 61 71 Page of 61 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 62 71 Page of 62 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt x
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert fatal message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
Page of 63 71 Page of 63 71
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
Page of 64 71 Page of 64 71
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 65 71 Page of 65 71
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt heavy_exclamation_mark
Page of 66 71 Page of 66 71
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert WARNING message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 67 71 Page of 67 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
Page of 68 71 Page of 68 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 69 71 Page of 69 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
Page of 70 71 Page of 70 71
Page of 71 71 Page of 71 71
13
413 Add13 an13 integraon13 name13 select13 an13 escalaon13 policy13 (use13 exisng13 or13 create13 a13 personalised13 policy13 via13 Configuraon13 agrave13 Escalaon13 Policies13 agrave13 New13 Escalaon13 Policy)13 select13 a13 noficaon13 urgency13 and13 tune13 the13 incident13 behaviour13 for13 your13 requirements13 13
513 Click13 Add13 Service13
13
The13 service13 is13 added13 and13 an13 integraon13 key13 is13 created13 to13 be13 used13 in13 the13 Logstash13 configuraon13
Page of 46 71 Page of 46 71
13
The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt be7eb1ea9c224018923e818d3e891d0b
incident_key =gt logstashhosttype
Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13
Page of 47 71 Page of 47 71
13
Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13
1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13
2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13
3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13
4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13
13 13 http
url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo
http_method =gt post
format =gt json
For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13
13 13 syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
Page of 48 71 Page of 48 71
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13
13
The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13
if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2
This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13
Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13
Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13
Logstash13 severity Shown13 in13 Omnibus13 as
alert Indeterminate
crical Warning
debug 7
emergency Clear
error Minor
informaonal 6
noce Crical
warning Major
Page of 49 71 Page of 49 71
13
Page of 50 71 Page of 50 71
Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13
input
tcp
port =gt 5000
type =gt syslog
filter
mutate
add_field =gt format =gt cf
grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
translate
field =gt data1
destination =gt data1
override =gt true
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
=== Check Alarm Criteria ===
Standard Key Words
if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
=== Alarm Exclusions ===
if [message] =~ (i)sometexthere
mutate remove_tag =gt servicename_alarm
Throttle alarms so we do not hit PD rate limits
Page of 52 71 Page of 52 71
if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 5
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle
after_count =gt 5
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 5
period =gt 1200
key =gt servicename_alarm_warning
add_tag =gt servicename_throttled
Page of 53 71 Page of 53 71
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
output
elasticsearch
hosts =gt localhost
stdout codec =gt rubydebug
if servicename_alarm_err in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
Page of 54 71 Page of 54 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 55 71 Page of 55 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
Page of 56 71 Page of 56 71
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert err message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_error in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
Page of 57 71 Page of 57 71
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 58 71 Page of 58 71
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 59 71 Page of 59 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert error message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
Page of 60 71 Page of 60 71
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
Page of 61 71 Page of 61 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 62 71 Page of 62 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt x
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert fatal message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
Page of 63 71 Page of 63 71
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
Page of 64 71 Page of 64 71
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 65 71 Page of 65 71
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt heavy_exclamation_mark
Page of 66 71 Page of 66 71
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert WARNING message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 67 71 Page of 67 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
Page of 68 71 Page of 68 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 69 71 Page of 69 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
Page of 70 71 Page of 70 71
Page of 71 71 Page of 71 71
13
The13 generated13 key13 can13 be13 used13 in13 the13 Logstash13 output13 pagerduty13 plugin13 using13 Logstash13 fields13 message13 the13 raw13 record13 data113 the13 Bluemix13 Appname13 etc13 from13 the13 grok13 output13 to13 customise13 the13 alerts13 sent13 to13 PagerDuty13
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt be7eb1ea9c224018923e818d3e891d0b
incident_key =gt logstashhosttype
Alerts13 sent13 from13 Logstash13 to13 Pagerduty13 appear13 similar13 to13 this13 one13 on13 the13 PagerDuty13 Incidents13 page13
Page of 47 71 Page of 47 71
13
Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13
1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13
2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13
3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13
4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13
13 13 http
url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo
http_method =gt post
format =gt json
For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13
13 13 syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
Page of 48 71 Page of 48 71
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13
13
The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13
if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2
This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13
Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13
Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13
Logstash13 severity Shown13 in13 Omnibus13 as
alert Indeterminate
crical Warning
debug 7
emergency Clear
error Minor
informaonal 6
noce Crical
warning Major
Page of 49 71 Page of 49 71
13
Page of 50 71 Page of 50 71
Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13
input
tcp
port =gt 5000
type =gt syslog
filter
mutate
add_field =gt format =gt cf
grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
translate
field =gt data1
destination =gt data1
override =gt true
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
=== Check Alarm Criteria ===
Standard Key Words
if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
=== Alarm Exclusions ===
if [message] =~ (i)sometexthere
mutate remove_tag =gt servicename_alarm
Throttle alarms so we do not hit PD rate limits
Page of 52 71 Page of 52 71
if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 5
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle
after_count =gt 5
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 5
period =gt 1200
key =gt servicename_alarm_warning
add_tag =gt servicename_throttled
Page of 53 71 Page of 53 71
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
output
elasticsearch
hosts =gt localhost
stdout codec =gt rubydebug
if servicename_alarm_err in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
Page of 54 71 Page of 54 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 55 71 Page of 55 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
Page of 56 71 Page of 56 71
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert err message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_error in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
Page of 57 71 Page of 57 71
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 58 71 Page of 58 71
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 59 71 Page of 59 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert error message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
Page of 60 71 Page of 60 71
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
Page of 61 71 Page of 61 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 62 71 Page of 62 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt x
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert fatal message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
Page of 63 71 Page of 63 71
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
Page of 64 71 Page of 64 71
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 65 71 Page of 65 71
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt heavy_exclamation_mark
Page of 66 71 Page of 66 71
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert WARNING message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 67 71 Page of 67 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
Page of 68 71 Page of 68 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 69 71 Page of 69 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
Page of 70 71 Page of 70 71
Page of 71 71 Page of 71 71
13
Integrate13 Logstash13 with13 IBM13 Netcool13 Omnibus13 To13 send13 generated13 alerts13 to13 IBM13 Netcool13 Omnibus13 complete13 these13 steps13
1 Use13 the13 syslog13 plugin13 or13 the13 hcp_plugin13 in13 the13 Logstash13 output13
2 Accept13 the13 alerts13 in13 IBM13 Netcool13 Omnibus13 using13 the13 Syslogd13 Probe13 or13 Hcp13 Probe13 and13 configuring13 them13 as13 required13
3 In13 the13 Logstash13 Output13 secon13 add13 plugin13 secons13 for13 either13 syslog13 or13 hcp13
4 Use13 these13 defaults13 for13 hcp13 and13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13
13 13 http
url =gt ldquohttpltOmnibusIPgtltOmnibusPortgtrdquo
http_method =gt post
format =gt json
For13 syslog13 add13 the13 values13 for13 the13 Omnibus13 IP13 address13 and13 port13 set13 the13 parameters13 similar13 to13 the13 following13 using13 Logstash13 fields13 message13 the13 raw13 record13 and13 data113 the13 Bluemix13 app13 name13 etc13 from13 the13 grok13 output13
13 13 syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
Page of 48 71 Page of 48 71
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13
13
The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13
if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2
This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13
Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13
Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13
Logstash13 severity Shown13 in13 Omnibus13 as
alert Indeterminate
crical Warning
debug 7
emergency Clear
error Minor
informaonal 6
noce Crical
warning Major
Page of 49 71 Page of 49 71
13
Page of 50 71 Page of 50 71
Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13
input
tcp
port =gt 5000
type =gt syslog
filter
mutate
add_field =gt format =gt cf
grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
translate
field =gt data1
destination =gt data1
override =gt true
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
=== Check Alarm Criteria ===
Standard Key Words
if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
=== Alarm Exclusions ===
if [message] =~ (i)sometexthere
mutate remove_tag =gt servicename_alarm
Throttle alarms so we do not hit PD rate limits
Page of 52 71 Page of 52 71
if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 5
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle
after_count =gt 5
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 5
period =gt 1200
key =gt servicename_alarm_warning
add_tag =gt servicename_throttled
Page of 53 71 Page of 53 71
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
output
elasticsearch
hosts =gt localhost
stdout codec =gt rubydebug
if servicename_alarm_err in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
Page of 54 71 Page of 54 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 55 71 Page of 55 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
Page of 56 71 Page of 56 71
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert err message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_error in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
Page of 57 71 Page of 57 71
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 58 71 Page of 58 71
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 59 71 Page of 59 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert error message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
Page of 60 71 Page of 60 71
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
Page of 61 71 Page of 61 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 62 71 Page of 62 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt x
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert fatal message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
Page of 63 71 Page of 63 71
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
Page of 64 71 Page of 64 71
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 65 71 Page of 65 71
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt heavy_exclamation_mark
Page of 66 71 Page of 66 71
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert WARNING message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 67 71 Page of 67 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
Page of 68 71 Page of 68 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 69 71 Page of 69 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
Page of 70 71 Page of 70 71
Page of 71 71 Page of 71 71
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
For13 the13 severity13 the13 pagerduty13 plugin13 has13 the13 following13 values13
13
The13 Omnibus13 instance13 in13 the13 scenariorsquos13 project13 environment13 has13 been13 set13 up13 to13 receive13 the13 severity13 as13 follows13
if ($Severity lt= 3) Severity = 5 else if ($Severity = 4) Severity = 4 else if ($Severity = 5) Severity = 3 else Severity = 2
This13 setup13 means13 the13 Logstash13 severies13 are13 translated13 into13 the13 Omnibus13 UI13 as13 follows13
Therefore13 you13 should13 select13 the13 Logstash13 severity13 based13 on13 the13 value13 users13 would13 expect13 to13 see13 in13 Omnibus13 for13 that13 specific13 alert13 condion13
Once13 alerts13 are13 being13 sent13 from13 Logstash13 to13 Omnibus13 they13 are13 shown13 as13 follows13
Logstash13 severity Shown13 in13 Omnibus13 as
alert Indeterminate
crical Warning
debug 7
emergency Clear
error Minor
informaonal 6
noce Crical
warning Major
Page of 49 71 Page of 49 71
13
Page of 50 71 Page of 50 71
Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13
input
tcp
port =gt 5000
type =gt syslog
filter
mutate
add_field =gt format =gt cf
grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
translate
field =gt data1
destination =gt data1
override =gt true
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
=== Check Alarm Criteria ===
Standard Key Words
if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
=== Alarm Exclusions ===
if [message] =~ (i)sometexthere
mutate remove_tag =gt servicename_alarm
Throttle alarms so we do not hit PD rate limits
Page of 52 71 Page of 52 71
if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 5
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle
after_count =gt 5
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 5
period =gt 1200
key =gt servicename_alarm_warning
add_tag =gt servicename_throttled
Page of 53 71 Page of 53 71
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
output
elasticsearch
hosts =gt localhost
stdout codec =gt rubydebug
if servicename_alarm_err in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
Page of 54 71 Page of 54 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 55 71 Page of 55 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
Page of 56 71 Page of 56 71
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert err message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_error in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
Page of 57 71 Page of 57 71
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 58 71 Page of 58 71
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 59 71 Page of 59 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert error message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
Page of 60 71 Page of 60 71
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
Page of 61 71 Page of 61 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 62 71 Page of 62 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt x
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert fatal message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
Page of 63 71 Page of 63 71
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
Page of 64 71 Page of 64 71
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 65 71 Page of 65 71
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt heavy_exclamation_mark
Page of 66 71 Page of 66 71
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert WARNING message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 67 71 Page of 67 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
Page of 68 71 Page of 68 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 69 71 Page of 69 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
Page of 70 71 Page of 70 71
Page of 71 71 Page of 71 71
13
Page of 50 71 Page of 50 71
Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13
input
tcp
port =gt 5000
type =gt syslog
filter
mutate
add_field =gt format =gt cf
grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
translate
field =gt data1
destination =gt data1
override =gt true
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
=== Check Alarm Criteria ===
Standard Key Words
if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
=== Alarm Exclusions ===
if [message] =~ (i)sometexthere
mutate remove_tag =gt servicename_alarm
Throttle alarms so we do not hit PD rate limits
Page of 52 71 Page of 52 71
if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 5
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle
after_count =gt 5
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 5
period =gt 1200
key =gt servicename_alarm_warning
add_tag =gt servicename_throttled
Page of 53 71 Page of 53 71
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
output
elasticsearch
hosts =gt localhost
stdout codec =gt rubydebug
if servicename_alarm_err in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
Page of 54 71 Page of 54 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 55 71 Page of 55 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
Page of 56 71 Page of 56 71
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert err message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_error in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
Page of 57 71 Page of 57 71
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 58 71 Page of 58 71
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 59 71 Page of 59 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert error message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
Page of 60 71 Page of 60 71
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
Page of 61 71 Page of 61 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 62 71 Page of 62 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt x
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert fatal message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
Page of 63 71 Page of 63 71
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
Page of 64 71 Page of 64 71
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 65 71 Page of 65 71
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt heavy_exclamation_mark
Page of 66 71 Page of 66 71
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert WARNING message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 67 71 Page of 67 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
Page of 68 71 Page of 68 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 69 71 Page of 69 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
Page of 70 71 Page of 70 71
Page of 71 71 Page of 71 71
Appendix13 Logstash13 configura0on13 Here13 is13 the13 full13 Logstash13 configuraon13 file13 used13 as13 part13 of13 the13 scenario13 project13 ELK13 instance13 for13 reference13
input
tcp
port =gt 5000
type =gt syslog
filter
mutate
add_field =gt format =gt cf
grok
match =gt message =gt NUMBERnum1 ltNUMBERnum2gtNUMBERnum3 TIMESTAMP_ISO8601tstamp WORDCFsource DATAdata1 [DATAdata2] NOTSPACEdash1 NOTSPACEdash2 GREEDYDATAmsgtosend
add_tag =gt [grokked]
end grok
translate
field =gt data1
destination =gt data1
override =gt true
dictionary =gt [ 13af977a-d2e4-4814-ab8c-29c9bef95724 voicemailshowservicemanagement
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
=== Check Alarm Criteria ===
Standard Key Words
if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
=== Alarm Exclusions ===
if [message] =~ (i)sometexthere
mutate remove_tag =gt servicename_alarm
Throttle alarms so we do not hit PD rate limits
Page of 52 71 Page of 52 71
if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 5
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle
after_count =gt 5
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 5
period =gt 1200
key =gt servicename_alarm_warning
add_tag =gt servicename_throttled
Page of 53 71 Page of 53 71
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
output
elasticsearch
hosts =gt localhost
stdout codec =gt rubydebug
if servicename_alarm_err in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
Page of 54 71 Page of 54 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 55 71 Page of 55 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
Page of 56 71 Page of 56 71
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert err message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_error in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
Page of 57 71 Page of 57 71
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 58 71 Page of 58 71
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 59 71 Page of 59 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert error message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
Page of 60 71 Page of 60 71
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
Page of 61 71 Page of 61 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 62 71 Page of 62 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt x
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert fatal message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
Page of 63 71 Page of 63 71
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
Page of 64 71 Page of 64 71
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 65 71 Page of 65 71
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt heavy_exclamation_mark
Page of 66 71 Page of 66 71
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert WARNING message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 67 71 Page of 67 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
Page of 68 71 Page of 68 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 69 71 Page of 69 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
Page of 70 71 Page of 70 71
Page of 71 71 Page of 71 71
f4c4bc34-856e-4867-9334-3dfd28365a9e microservices-catalogapi-rstoner-1615
059ba88b-0e42-4e29-858a-5ebd5762389c Microservices-UI-rstoner-1639
9b8ceb56-d952-40f7-9d84-aa06ea4d2dbe Microservices-OrdersAPI-rstoner-1622 ]
add_tag =gt translated
=== Check Alarm Criteria ===
Standard Key Words
if [message] =~ (err)
mutate add_tag =gt servicename_alarm_err
if [message] =~ (Error)
mutate add_tag =gt servicename_alarm_error
if [message] =~ (fatal)
mutate add_tag =gt servicename_alarm_fatal
if [message] =~ (WARNING)
mutate add_tag =gt servicename_alarm_warning
if [message] =~ (STOPPED|Stopped|CRASHED)
mutate add_tag =gt servicename_alarm_stopped
=== Alarm Exclusions ===
if [message] =~ (i)sometexthere
mutate remove_tag =gt servicename_alarm
Throttle alarms so we do not hit PD rate limits
Page of 52 71 Page of 52 71
if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 5
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle
after_count =gt 5
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 5
period =gt 1200
key =gt servicename_alarm_warning
add_tag =gt servicename_throttled
Page of 53 71 Page of 53 71
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
output
elasticsearch
hosts =gt localhost
stdout codec =gt rubydebug
if servicename_alarm_err in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
Page of 54 71 Page of 54 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 55 71 Page of 55 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
Page of 56 71 Page of 56 71
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert err message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_error in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
Page of 57 71 Page of 57 71
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 58 71 Page of 58 71
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 59 71 Page of 59 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert error message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
Page of 60 71 Page of 60 71
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
Page of 61 71 Page of 61 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 62 71 Page of 62 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt x
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert fatal message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
Page of 63 71 Page of 63 71
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
Page of 64 71 Page of 64 71
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 65 71 Page of 65 71
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt heavy_exclamation_mark
Page of 66 71 Page of 66 71
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert WARNING message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 67 71 Page of 67 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
Page of 68 71 Page of 68 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 69 71 Page of 69 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
Page of 70 71 Page of 70 71
Page of 71 71 Page of 71 71
if servicename_alarm_err in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_err
add_tag =gt servicename_throttled
if servicename_alarm_error in [tags]
throttle
after_count =gt 5
period =gt 600
key =gt servicename_alarm_error
add_tag =gt servicename_throttled
if servicename_alarm_fatal in [tags]
throttle
after_count =gt 5
period =gt 900
key =gt servicename_alarm_fatal
add_tag =gt servicename_throttled
if servicename_alarm_warning in [tags]
throttle
after_count =gt 5
period =gt 1200
key =gt servicename_alarm_warning
add_tag =gt servicename_throttled
Page of 53 71 Page of 53 71
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
output
elasticsearch
hosts =gt localhost
stdout codec =gt rubydebug
if servicename_alarm_err in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
Page of 54 71 Page of 54 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 55 71 Page of 55 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
Page of 56 71 Page of 56 71
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert err message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_error in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
Page of 57 71 Page of 57 71
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 58 71 Page of 58 71
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 59 71 Page of 59 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert error message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
Page of 60 71 Page of 60 71
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
Page of 61 71 Page of 61 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 62 71 Page of 62 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt x
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert fatal message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
Page of 63 71 Page of 63 71
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
Page of 64 71 Page of 64 71
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 65 71 Page of 65 71
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt heavy_exclamation_mark
Page of 66 71 Page of 66 71
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert WARNING message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 67 71 Page of 67 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
Page of 68 71 Page of 68 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 69 71 Page of 69 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
Page of 70 71 Page of 70 71
Page of 71 71 Page of 71 71
if servicename_alarm_stopped in [tags]
throttle
after_count =gt 5
period =gt 300
key =gt servicename_alarm_stopped
add_tag =gt servicename_throttled
output
elasticsearch
hosts =gt localhost
stdout codec =gt rubydebug
if servicename_alarm_err in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
Page of 54 71 Page of 54 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 55 71 Page of 55 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
Page of 56 71 Page of 56 71
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert err message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_error in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
Page of 57 71 Page of 57 71
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 58 71 Page of 58 71
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 59 71 Page of 59 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert error message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
Page of 60 71 Page of 60 71
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
Page of 61 71 Page of 61 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 62 71 Page of 62 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt x
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert fatal message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
Page of 63 71 Page of 63 71
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
Page of 64 71 Page of 64 71
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 65 71 Page of 65 71
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt heavy_exclamation_mark
Page of 66 71 Page of 66 71
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert WARNING message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 67 71 Page of 67 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
Page of 68 71 Page of 68 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 69 71 Page of 69 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
Page of 70 71 Page of 70 71
Page of 71 71 Page of 71 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 55 71 Page of 55 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
Page of 56 71 Page of 56 71
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert err message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_error in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
Page of 57 71 Page of 57 71
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 58 71 Page of 58 71
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 59 71 Page of 59 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert error message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
Page of 60 71 Page of 60 71
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
Page of 61 71 Page of 61 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 62 71 Page of 62 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt x
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert fatal message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
Page of 63 71 Page of 63 71
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
Page of 64 71 Page of 64 71
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 65 71 Page of 65 71
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt heavy_exclamation_mark
Page of 66 71 Page of 66 71
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert WARNING message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 67 71 Page of 67 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
Page of 68 71 Page of 68 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 69 71 Page of 69 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
Page of 70 71 Page of 70 71
Page of 71 71 Page of 71 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
Page of 56 71 Page of 56 71
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert err message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_error in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
Page of 57 71 Page of 57 71
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 58 71 Page of 58 71
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 59 71 Page of 59 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert error message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
Page of 60 71 Page of 60 71
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
Page of 61 71 Page of 61 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 62 71 Page of 62 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt x
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert fatal message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
Page of 63 71 Page of 63 71
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
Page of 64 71 Page of 64 71
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 65 71 Page of 65 71
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt heavy_exclamation_mark
Page of 66 71 Page of 66 71
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert WARNING message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 67 71 Page of 67 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
Page of 68 71 Page of 68 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 69 71 Page of 69 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
Page of 70 71 Page of 70 71
Page of 71 71 Page of 71 71
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert err message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_error in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
Page of 57 71 Page of 57 71
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 58 71 Page of 58 71
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 59 71 Page of 59 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert error message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
Page of 60 71 Page of 60 71
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
Page of 61 71 Page of 61 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 62 71 Page of 62 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt x
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert fatal message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
Page of 63 71 Page of 63 71
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
Page of 64 71 Page of 64 71
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 65 71 Page of 65 71
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt heavy_exclamation_mark
Page of 66 71 Page of 66 71
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert WARNING message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 67 71 Page of 67 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
Page of 68 71 Page of 68 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 69 71 Page of 69 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
Page of 70 71 Page of 70 71
Page of 71 71 Page of 71 71
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 58 71 Page of 58 71
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 59 71 Page of 59 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert error message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
Page of 60 71 Page of 60 71
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
Page of 61 71 Page of 61 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 62 71 Page of 62 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt x
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert fatal message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
Page of 63 71 Page of 63 71
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
Page of 64 71 Page of 64 71
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 65 71 Page of 65 71
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt heavy_exclamation_mark
Page of 66 71 Page of 66 71
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert WARNING message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 67 71 Page of 67 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
Page of 68 71 Page of 68 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 69 71 Page of 69 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
Page of 70 71 Page of 70 71
Page of 71 71 Page of 71 71
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 59 71 Page of 59 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert error message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
Page of 60 71 Page of 60 71
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
Page of 61 71 Page of 61 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 62 71 Page of 62 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt x
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert fatal message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
Page of 63 71 Page of 63 71
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
Page of 64 71 Page of 64 71
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 65 71 Page of 65 71
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt heavy_exclamation_mark
Page of 66 71 Page of 66 71
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert WARNING message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 67 71 Page of 67 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
Page of 68 71 Page of 68 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 69 71 Page of 69 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
Page of 70 71 Page of 70 71
Page of 71 71 Page of 71 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert error message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt warning
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
Page of 60 71 Page of 60 71
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
Page of 61 71 Page of 61 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 62 71 Page of 62 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt x
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert fatal message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
Page of 63 71 Page of 63 71
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
Page of 64 71 Page of 64 71
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 65 71 Page of 65 71
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt heavy_exclamation_mark
Page of 66 71 Page of 66 71
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert WARNING message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 67 71 Page of 67 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
Page of 68 71 Page of 68 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 69 71 Page of 69 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
Page of 70 71 Page of 70 71
Page of 71 71 Page of 71 71
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_fatal in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
Page of 61 71 Page of 61 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 62 71 Page of 62 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt x
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert fatal message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
Page of 63 71 Page of 63 71
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
Page of 64 71 Page of 64 71
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 65 71 Page of 65 71
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt heavy_exclamation_mark
Page of 66 71 Page of 66 71
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert WARNING message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 67 71 Page of 67 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
Page of 68 71 Page of 68 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 69 71 Page of 69 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
Page of 70 71 Page of 70 71
Page of 71 71 Page of 71 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 62 71 Page of 62 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt x
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert fatal message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
Page of 63 71 Page of 63 71
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
Page of 64 71 Page of 64 71
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 65 71 Page of 65 71
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt heavy_exclamation_mark
Page of 66 71 Page of 66 71
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert WARNING message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 67 71 Page of 67 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
Page of 68 71 Page of 68 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 69 71 Page of 69 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
Page of 70 71 Page of 70 71
Page of 71 71 Page of 71 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt x
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert fatal message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
Page of 63 71 Page of 63 71
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
Page of 64 71 Page of 64 71
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 65 71 Page of 65 71
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt heavy_exclamation_mark
Page of 66 71 Page of 66 71
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert WARNING message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 67 71 Page of 67 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
Page of 68 71 Page of 68 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 69 71 Page of 69 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
Page of 70 71 Page of 70 71
Page of 71 71 Page of 71 71
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_warning in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
Page of 64 71 Page of 64 71
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 65 71 Page of 65 71
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt heavy_exclamation_mark
Page of 66 71 Page of 66 71
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert WARNING message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 67 71 Page of 67 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
Page of 68 71 Page of 68 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 69 71 Page of 69 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
Page of 70 71 Page of 70 71
Page of 71 71 Page of 71 71
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
Page of 65 71 Page of 65 71
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt heavy_exclamation_mark
Page of 66 71 Page of 66 71
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert WARNING message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 67 71 Page of 67 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
Page of 68 71 Page of 68 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 69 71 Page of 69 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
Page of 70 71 Page of 70 71
Page of 71 71 Page of 71 71
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt heavy_exclamation_mark
Page of 66 71 Page of 66 71
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert WARNING message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 67 71 Page of 67 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
Page of 68 71 Page of 68 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 69 71 Page of 69 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
Page of 70 71 Page of 70 71
Page of 71 71 Page of 71 71
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert WARNING message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt critical
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
if servicename_alarm_stopped in [tags] and servicename_throttled not in [tags]
if [data1] == voicemailshowservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
Page of 67 71 Page of 67 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
Page of 68 71 Page of 68 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 69 71 Page of 69 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
Page of 70 71 Page of 70 71
Page of 71 71 Page of 71 71
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == monverifyservicemanagement
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == ms-catalog-postpyretic-pomiculture
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == microservices-catalogapi-rstoner-1615
Page of 68 71 Page of 68 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 69 71 Page of 69 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
Page of 70 71 Page of 70 71
Page of 71 71 Page of 71 71
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-UI-rstoner-1639
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
if [data1] == Microservices-OrdersAPI-rstoner-1622
pagerduty
event_type =gt trigger
description =gt data1 alert reported on host
details =gt
timestamp =gt timestamp
message =gt message
Page of 69 71 Page of 69 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
Page of 70 71 Page of 70 71
Page of 71 71 Page of 71 71
service_key =gt ltpagerduty_service_keygt
incident_key =gt logstashhosttype
slack
url =gt ltSlackIntegrationURLgt
channel =gt slack_channel
username =gt slack_user
icon_emoji =gt interrobang
icon_url =gt [icon url would be overriden by icon_emoji - optional]
format =gt ELK alert Stopped message reported for data1 on host message
attachments =gt
http
url =gt httpltOmnibusIPgtltOmnibusPortgt
http_method =gt post
format =gt json
syslog
facility =gt log alert
host =gt ltOmnibusIPgt
port =gt ltOmnibusPortgt
severity =gt notice
appname =gt logstash
message =gt msgtosend
msgid =gt tstamp
procid =gt data1
rfc =gt rfc5424
sourcehost =gt host
Page of 70 71 Page of 70 71
Page of 71 71 Page of 71 71
Page of 71 71 Page of 71 71