hqs-ugm2016-improved-log-analysis-with-elk - · pdf fileno more logstash enhancement:...
TRANSCRIPT
![Page 1: HQS-UGM2016-Improved-Log-Analysis-with-ELK - · PDF fileNo more Logstash Enhancement: Introduction to additional logging information GeoIP in ODS Log? ... Microsoft PowerPoint - Improved](https://reader034.vdocuments.net/reader034/viewer/2022051600/5a9fa0447f8b9a6c178cf9e7/html5/thumbnails/1.jpg)
Improved ODS Analysis with ELKOr how to quickly parse your log files
HighQSoft GmbH | www.highqsoft.de | 11.05.2016
1
Alexander Ziller / Constantin Badescu
![Page 2: HQS-UGM2016-Improved-Log-Analysis-with-ELK - · PDF fileNo more Logstash Enhancement: Introduction to additional logging information GeoIP in ODS Log? ... Microsoft PowerPoint - Improved](https://reader034.vdocuments.net/reader034/viewer/2022051600/5a9fa0447f8b9a6c178cf9e7/html5/thumbnails/2.jpg)
HighQSoft GmbH | www.highqsoft.de | 11.05.2016
2
Content
Traditional situationDefining an Solution Approach: What is ELK?Architecture of ELKIntegration with ODS systemsUse-Cases of ELK and ODS
![Page 3: HQS-UGM2016-Improved-Log-Analysis-with-ELK - · PDF fileNo more Logstash Enhancement: Introduction to additional logging information GeoIP in ODS Log? ... Microsoft PowerPoint - Improved](https://reader034.vdocuments.net/reader034/viewer/2022051600/5a9fa0447f8b9a6c178cf9e7/html5/thumbnails/3.jpg)
HighQSoft GmbH | www.highqsoft.de | 11.05.2016
3
Traditional situationLogs don’t come to you. You need to get to them.
RDP
Putty
Putty, VNC
Avalon CFS Avalon CFS
User
Avalon Apache HTTP Syslog …
Avalon Apache HTTP Syslog …
Tomcat Importer Tomcat Importer
This might be just one system. You may want to cover multiple.
LinuxLinux
Windows
![Page 4: HQS-UGM2016-Improved-Log-Analysis-with-ELK - · PDF fileNo more Logstash Enhancement: Introduction to additional logging information GeoIP in ODS Log? ... Microsoft PowerPoint - Improved](https://reader034.vdocuments.net/reader034/viewer/2022051600/5a9fa0447f8b9a6c178cf9e7/html5/thumbnails/4.jpg)
HighQSoft GmbH | www.highqsoft.de | 11.05.2016
4
Defining an Solution ApproachWhat is ELK?
ElasticSearch – Logstash – Kibana
Elasticsearch: Indexer with scalability
Logstash: Logserver with parsing capabilities
Kibana: WEB Frontend for Logstash
Beats: Log-Forwarder for Logstash
FELK: Filebeat – Elasticsearch – Logstash – Kibana
![Page 5: HQS-UGM2016-Improved-Log-Analysis-with-ELK - · PDF fileNo more Logstash Enhancement: Introduction to additional logging information GeoIP in ODS Log? ... Microsoft PowerPoint - Improved](https://reader034.vdocuments.net/reader034/viewer/2022051600/5a9fa0447f8b9a6c178cf9e7/html5/thumbnails/5.jpg)
HighQSoft GmbH | www.highqsoft.de | 11.05.2016
5
Architecture of ELK
Avalon Apache HTTP Syslog …
Avalon Apache HTTP Syslog …
Tomcat Tomcat
Avalon … Avalon …
Beat Beat
Beat Beat
Logstash Logstash
ElasticSearch ElasticSearch
Kibana Kibana
![Page 6: HQS-UGM2016-Improved-Log-Analysis-with-ELK - · PDF fileNo more Logstash Enhancement: Introduction to additional logging information GeoIP in ODS Log? ... Microsoft PowerPoint - Improved](https://reader034.vdocuments.net/reader034/viewer/2022051600/5a9fa0447f8b9a6c178cf9e7/html5/thumbnails/6.jpg)
HighQSoft GmbH | www.highqsoft.de | 11.05.2016
6
Integration of ODS logsGathering of the log data
...20160426 13:43:01.293535 diff: 0 Thread 7f2dffea7700 Session 851 AoSession_GetContextByName()...
Avalon Apache HTTP Syslog …
+Beat
Avalon Apache HTTP Syslog …
+Beat
![Page 7: HQS-UGM2016-Improved-Log-Analysis-with-ELK - · PDF fileNo more Logstash Enhancement: Introduction to additional logging information GeoIP in ODS Log? ... Microsoft PowerPoint - Improved](https://reader034.vdocuments.net/reader034/viewer/2022051600/5a9fa0447f8b9a6c178cf9e7/html5/thumbnails/7.jpg)
HighQSoft GmbH | www.highqsoft.de | 11.05.2016
7
Integration with ODSParsing of the log data
20160426 13:43:01.293535 diff: 0 Thread 7f2dffea7700 Session 851 AoSession_GetContextByName()
Date Thread ID Session ID ODS Object+Method
Message
…+
Logstash Elasticsearch Kibana
…+
Logstash Elasticsearch Kibana
![Page 8: HQS-UGM2016-Improved-Log-Analysis-with-ELK - · PDF fileNo more Logstash Enhancement: Introduction to additional logging information GeoIP in ODS Log? ... Microsoft PowerPoint - Improved](https://reader034.vdocuments.net/reader034/viewer/2022051600/5a9fa0447f8b9a6c178cf9e7/html5/thumbnails/8.jpg)
HighQSoft GmbH | www.highqsoft.de | 11.05.2016
8
Integration with ODSProviding the log data
…+
Logstash Elasticsearch Kibana
…+
Logstash Elasticsearch Kibana
![Page 9: HQS-UGM2016-Improved-Log-Analysis-with-ELK - · PDF fileNo more Logstash Enhancement: Introduction to additional logging information GeoIP in ODS Log? ... Microsoft PowerPoint - Improved](https://reader034.vdocuments.net/reader034/viewer/2022051600/5a9fa0447f8b9a6c178cf9e7/html5/thumbnails/9.jpg)
HighQSoft GmbH | www.highqsoft.de | 11.05.2016
9
Integration with ODS
License AdministrationIntegration of multiple RLM servers
Availability of combined Reportlogs
Currently: Standard-Level
> License Utilization (License/time)
Analyzing the log data: Use-Cases of ELK and ODS …
+ Logstash Elasticsearch Kibana
…+
Logstash Elasticsearch Kibana
![Page 10: HQS-UGM2016-Improved-Log-Analysis-with-ELK - · PDF fileNo more Logstash Enhancement: Introduction to additional logging information GeoIP in ODS Log? ... Microsoft PowerPoint - Improved](https://reader034.vdocuments.net/reader034/viewer/2022051600/5a9fa0447f8b9a6c178cf9e7/html5/thumbnails/10.jpg)
HighQSoft GmbH | www.highqsoft.de | 11.05.2016
10
Use-Cases of ELK and ODS
Avalon Session CountingIntegration of multiple Avalon Server
Availability of ODS Logs
Currently: Debug-Level 3
>> Sessions by Avalon Instance?
>> Sessions by User(Group)?
>> Number of Log-Ins?
Analyzing the log data: Use-Cases of ELK and ODS …
+ Logstash Elasticsearch Kibana
…+
Logstash Elasticsearch Kibana
![Page 11: HQS-UGM2016-Improved-Log-Analysis-with-ELK - · PDF fileNo more Logstash Enhancement: Introduction to additional logging information GeoIP in ODS Log? ... Microsoft PowerPoint - Improved](https://reader034.vdocuments.net/reader034/viewer/2022051600/5a9fa0447f8b9a6c178cf9e7/html5/thumbnails/11.jpg)
HighQSoft GmbH | www.highqsoft.de | 11.05.2016
11
Use-Cases of ELK and ODS
Further Use-Cases:Error searching
Message analysis
…
Analyzing the log data: Use-Cases of ELK and ODS …
+ Logstash Elasticsearch Kibana
…+
Logstash Elasticsearch Kibana
![Page 12: HQS-UGM2016-Improved-Log-Analysis-with-ELK - · PDF fileNo more Logstash Enhancement: Introduction to additional logging information GeoIP in ODS Log? ... Microsoft PowerPoint - Improved](https://reader034.vdocuments.net/reader034/viewer/2022051600/5a9fa0447f8b9a6c178cf9e7/html5/thumbnails/12.jpg)
HighQSoft GmbH | www.highqsoft.de | 11.05.2016
12
Use-Cases of ELK and ODSAnalyzing the log data: Demonstration
![Page 13: HQS-UGM2016-Improved-Log-Analysis-with-ELK - · PDF fileNo more Logstash Enhancement: Introduction to additional logging information GeoIP in ODS Log? ... Microsoft PowerPoint - Improved](https://reader034.vdocuments.net/reader034/viewer/2022051600/5a9fa0447f8b9a6c178cf9e7/html5/thumbnails/13.jpg)
HighQSoft GmbH | www.highqsoft.de | 11.05.2016
13
What is the current statusInstalling and configuring setup
FELK stack available for Windows/LinuxPartial Repository support for LinuxConfiguration files: YAML, JSON
Parsing patterns depend on the log formattingCategorize logs for easy browsingFELK can be extended with “Shield” for SecurityAble to index multiple weeks/months of ODS logs
![Page 14: HQS-UGM2016-Improved-Log-Analysis-with-ELK - · PDF fileNo more Logstash Enhancement: Introduction to additional logging information GeoIP in ODS Log? ... Microsoft PowerPoint - Improved](https://reader034.vdocuments.net/reader034/viewer/2022051600/5a9fa0447f8b9a6c178cf9e7/html5/thumbnails/14.jpg)
HighQSoft GmbH | www.highqsoft.de | 11.05.2016
14
What is next for ELK and ODS?Expanding ELK to your needs
Simplification: Avalon as a BeatDirect integration into ELK, less configuration
Simplification: Avalon as a Log-Provider (with Avalon Service?)No more Logstash
Enhancement: Introduction to additional logging informationGeoIP in ODS Log?
Standardization: Standard ODS log patterns and parsersAny Log-Level SupportModelMapper Compatibility (own logging-rules)Avalon Suite 2017 Integration (integrated service)
Intelligent Analysis: Additional plug-ins for Kibana to allow business-log-icODS related information merged with log data
![Page 15: HQS-UGM2016-Improved-Log-Analysis-with-ELK - · PDF fileNo more Logstash Enhancement: Introduction to additional logging information GeoIP in ODS Log? ... Microsoft PowerPoint - Improved](https://reader034.vdocuments.net/reader034/viewer/2022051600/5a9fa0447f8b9a6c178cf9e7/html5/thumbnails/15.jpg)
HighQSoft GmbH | www.highqsoft.de | 11.05.2016
15
Thank you
We hope that the presentation helps you to manage your log-files!
HighQSoft GmbHSchloßborner Weg 6b61479 GlashüttenGermany
Alexander Ziller+49 6147 [email protected]