Download - Logonomics
![Page 1: Logonomics](https://reader033.vdocuments.net/reader033/viewer/2022051612/54c1bc3d4a795955738b45e0/html5/thumbnails/1.jpg)
LOGONOMICS: The Hidden Side of Blackboard Logs by steve feldman @PerfForensics
![Page 2: Logonomics](https://reader033.vdocuments.net/reader033/viewer/2022051612/54c1bc3d4a795955738b45e0/html5/thumbnails/2.jpg)
Logging Doesn’t Suck
![Page 3: Logonomics](https://reader033.vdocuments.net/reader033/viewer/2022051612/54c1bc3d4a795955738b45e0/html5/thumbnails/3.jpg)
![Page 4: Logonomics](https://reader033.vdocuments.net/reader033/viewer/2022051612/54c1bc3d4a795955738b45e0/html5/thumbnails/4.jpg)
![Page 5: Logonomics](https://reader033.vdocuments.net/reader033/viewer/2022051612/54c1bc3d4a795955738b45e0/html5/thumbnails/5.jpg)
It’s Like Fishing in the Night…
![Page 6: Logonomics](https://reader033.vdocuments.net/reader033/viewer/2022051612/54c1bc3d4a795955738b45e0/html5/thumbnails/6.jpg)
![Page 7: Logonomics](https://reader033.vdocuments.net/reader033/viewer/2022051612/54c1bc3d4a795955738b45e0/html5/thumbnails/7.jpg)
![Page 8: Logonomics](https://reader033.vdocuments.net/reader033/viewer/2022051612/54c1bc3d4a795955738b45e0/html5/thumbnails/8.jpg)
So Why Don’t We Talk About Logs More OJen?
![Page 9: Logonomics](https://reader033.vdocuments.net/reader033/viewer/2022051612/54c1bc3d4a795955738b45e0/html5/thumbnails/9.jpg)
At least 20% of all people in this room don’t know where to find their logs.
![Page 10: Logonomics](https://reader033.vdocuments.net/reader033/viewer/2022051612/54c1bc3d4a795955738b45e0/html5/thumbnails/10.jpg)
At least 50% of all people in this room don’t look at their logs.
![Page 11: Logonomics](https://reader033.vdocuments.net/reader033/viewer/2022051612/54c1bc3d4a795955738b45e0/html5/thumbnails/11.jpg)
At least 60% of all people in this room don’t visualize their log data.
![Page 12: Logonomics](https://reader033.vdocuments.net/reader033/viewer/2022051612/54c1bc3d4a795955738b45e0/html5/thumbnails/12.jpg)
At least 75% of all people in this room don’t correlate data between logs.
![Page 13: Logonomics](https://reader033.vdocuments.net/reader033/viewer/2022051612/54c1bc3d4a795955738b45e0/html5/thumbnails/13.jpg)
At least 90% of all people in this room don’t standardize the management of
logs to a centralized service.
![Page 14: Logonomics](https://reader033.vdocuments.net/reader033/viewer/2022051612/54c1bc3d4a795955738b45e0/html5/thumbnails/14.jpg)
At least 95% of all people in this room don’t alert IT staff based on a specific
log event.
![Page 15: Logonomics](https://reader033.vdocuments.net/reader033/viewer/2022051612/54c1bc3d4a795955738b45e0/html5/thumbnails/15.jpg)
If a System Doesn’t Output to a Log Do We Assume Nobody is Using it?
![Page 16: Logonomics](https://reader033.vdocuments.net/reader033/viewer/2022051612/54c1bc3d4a795955738b45e0/html5/thumbnails/16.jpg)
If a System ConZnuously Spews Data to a Log Do We Ignore it?
![Page 17: Logonomics](https://reader033.vdocuments.net/reader033/viewer/2022051612/54c1bc3d4a795955738b45e0/html5/thumbnails/17.jpg)
![Page 18: Logonomics](https://reader033.vdocuments.net/reader033/viewer/2022051612/54c1bc3d4a795955738b45e0/html5/thumbnails/18.jpg)
What We Can Do With Our Log Data LOGONOMICS: The Hidden Side of Blackboard Logs
![Page 19: Logonomics](https://reader033.vdocuments.net/reader033/viewer/2022051612/54c1bc3d4a795955738b45e0/html5/thumbnails/19.jpg)
Trending and Intelligence
Service Levels
Threats and VulnerabiliZes
Responsiveness Reliability
![Page 20: Logonomics](https://reader033.vdocuments.net/reader033/viewer/2022051612/54c1bc3d4a795955738b45e0/html5/thumbnails/20.jpg)
Primer Data Points Everyone Should Know
Unique Requests
Time Series of Requests
ConcentraZon of Request Types
Origin of Requests
Quick Averages
Cascading Issues Across Logs
![Page 21: Logonomics](https://reader033.vdocuments.net/reader033/viewer/2022051612/54c1bc3d4a795955738b45e0/html5/thumbnails/21.jpg)
Combining Other Data with Log Data
CorrelaZon
Root Cause
InterpretaZon
CompleZon of Message
Full Picture Sequence and Timelines
![Page 22: Logonomics](https://reader033.vdocuments.net/reader033/viewer/2022051612/54c1bc3d4a795955738b45e0/html5/thumbnails/22.jpg)
Types of Data We Can Get LOGONOMICS: The Hidden Side of Blackboard Logs
![Page 23: Logonomics](https://reader033.vdocuments.net/reader033/viewer/2022051612/54c1bc3d4a795955738b45e0/html5/thumbnails/23.jpg)
Business AnalyZcs: AdopZon and Growth
![Page 24: Logonomics](https://reader033.vdocuments.net/reader033/viewer/2022051612/54c1bc3d4a795955738b45e0/html5/thumbnails/24.jpg)
System Health
![Page 25: Logonomics](https://reader033.vdocuments.net/reader033/viewer/2022051612/54c1bc3d4a795955738b45e0/html5/thumbnails/25.jpg)
Capacity Planning
![Page 26: Logonomics](https://reader033.vdocuments.net/reader033/viewer/2022051612/54c1bc3d4a795955738b45e0/html5/thumbnails/26.jpg)
Security and Threat Analysis
![Page 27: Logonomics](https://reader033.vdocuments.net/reader033/viewer/2022051612/54c1bc3d4a795955738b45e0/html5/thumbnails/27.jpg)
Quality and Experience: MeeZng SLAs
![Page 28: Logonomics](https://reader033.vdocuments.net/reader033/viewer/2022051612/54c1bc3d4a795955738b45e0/html5/thumbnails/28.jpg)
Replay and Benchmarking
![Page 29: Logonomics](https://reader033.vdocuments.net/reader033/viewer/2022051612/54c1bc3d4a795955738b45e0/html5/thumbnails/29.jpg)
Insight into the BbLogs LOGONOMICS: The Hidden Side of Blackboard Logs
![Page 30: Logonomics](https://reader033.vdocuments.net/reader033/viewer/2022051612/54c1bc3d4a795955738b45e0/html5/thumbnails/30.jpg)
Four Horseman of Logs
![Page 31: Logonomics](https://reader033.vdocuments.net/reader033/viewer/2022051612/54c1bc3d4a795955738b45e0/html5/thumbnails/31.jpg)
Bablefield of Other Logs
• AuthenZcaZon • Plugins Directory • NauZlus for events • Monitoring (System Logs) – Syslogs and Rsyslogs (/var/messages) – Windows Event Logs
![Page 32: Logonomics](https://reader033.vdocuments.net/reader033/viewer/2022051612/54c1bc3d4a795955738b45e0/html5/thumbnails/32.jpg)
Is there a Most Important Log?
![Page 33: Logonomics](https://reader033.vdocuments.net/reader033/viewer/2022051612/54c1bc3d4a795955738b45e0/html5/thumbnails/33.jpg)
Access Log
Log Formafng Mabers Log Levels (INFO, WARN, ERROR)
mod_log_forensic
Use %k, %T and %D
Decompose the URI
Log Formafng Mabers
![Page 34: Logonomics](https://reader033.vdocuments.net/reader033/viewer/2022051612/54c1bc3d4a795955738b45e0/html5/thumbnails/34.jpg)
Is there a 2nd Most Important Log?
![Page 35: Logonomics](https://reader033.vdocuments.net/reader033/viewer/2022051612/54c1bc3d4a795955738b45e0/html5/thumbnails/35.jpg)
Tomcat and Java Logs
Stack Traces Startup OpZons
GC Events
GC Pauses and Status
![Page 36: Logonomics](https://reader033.vdocuments.net/reader033/viewer/2022051612/54c1bc3d4a795955738b45e0/html5/thumbnails/36.jpg)
Tools We Should Consider LOGONOMICS: The Hidden Side of Blackboard Logs
![Page 37: Logonomics](https://reader033.vdocuments.net/reader033/viewer/2022051612/54c1bc3d4a795955738b45e0/html5/thumbnails/37.jpg)
It’s All About the Right Fishing Rod
![Page 38: Logonomics](https://reader033.vdocuments.net/reader033/viewer/2022051612/54c1bc3d4a795955738b45e0/html5/thumbnails/38.jpg)
CAT!
GREP!
TAIL!
SED!AWK!
SORT!
![Page 39: Logonomics](https://reader033.vdocuments.net/reader033/viewer/2022051612/54c1bc3d4a795955738b45e0/html5/thumbnails/39.jpg)
GROK!
![Page 40: Logonomics](https://reader033.vdocuments.net/reader033/viewer/2022051612/54c1bc3d4a795955738b45e0/html5/thumbnails/40.jpg)
SomeZmes a Net is Beber to Cast
![Page 41: Logonomics](https://reader033.vdocuments.net/reader033/viewer/2022051612/54c1bc3d4a795955738b45e0/html5/thumbnails/41.jpg)
Log CentralizaZon
![Page 42: Logonomics](https://reader033.vdocuments.net/reader033/viewer/2022051612/54c1bc3d4a795955738b45e0/html5/thumbnails/42.jpg)
Please Take All My Logs
Format Lots of Log Data
Send it Down the River
![Page 43: Logonomics](https://reader033.vdocuments.net/reader033/viewer/2022051612/54c1bc3d4a795955738b45e0/html5/thumbnails/43.jpg)
![Page 44: Logonomics](https://reader033.vdocuments.net/reader033/viewer/2022051612/54c1bc3d4a795955738b45e0/html5/thumbnails/44.jpg)
• amqp • exec • file • gelf • redis • stdin • stomp • syslog • tcp • twiber • xmpp • zeromq
• amqp • elasZcsearch • elasZcsearch_
river • file • ganglia • gelf • graphite • internal • loggly • mongodb • nagios
• date • dns • gelfify • grep • grok • grokdisco
very • json • mulZline • mutate • split
• null • redis • statsd • stdout • stomp • tcp • websocket • xmpp • zabbix • zeromq
Inputs Filters Outputs
![Page 45: Logonomics](https://reader033.vdocuments.net/reader033/viewer/2022051612/54c1bc3d4a795955738b45e0/html5/thumbnails/45.jpg)
Configure Apache for JSON log
• hbp://cookbook.logstash.net/recipes/apache-‐json-‐logs/
![Page 46: Logonomics](https://reader033.vdocuments.net/reader033/viewer/2022051612/54c1bc3d4a795955738b45e0/html5/thumbnails/46.jpg)
Configure Tomcat for MulZ-‐Line Filter
![Page 47: Logonomics](https://reader033.vdocuments.net/reader033/viewer/2022051612/54c1bc3d4a795955738b45e0/html5/thumbnails/47.jpg)
Setup Bb to feed logstash
![Page 48: Logonomics](https://reader033.vdocuments.net/reader033/viewer/2022051612/54c1bc3d4a795955738b45e0/html5/thumbnails/48.jpg)
![Page 49: Logonomics](https://reader033.vdocuments.net/reader033/viewer/2022051612/54c1bc3d4a795955738b45e0/html5/thumbnails/49.jpg)
What We Use Logstash
Log AggregaZon
Non-‐FuncZonal Requirements
Event NoZficaZon
IntegraZon with Zabbix
Kibana Front-‐End Redis Inputs & Outputs
Indexing
![Page 50: Logonomics](https://reader033.vdocuments.net/reader033/viewer/2022051612/54c1bc3d4a795955738b45e0/html5/thumbnails/50.jpg)
Simple Challenge to All
• Setup Logstash architecture (All Single Node) • Start shipping basic log files – Apache 2.X access log or IIS web server log – Tomcat Catalina log file
• Output results to statsD (Etsy Project) – Simple Use Case: IncremenZng HTTP codes (200, 300, 400)
• Visualize statsD data with Graphite
![Page 51: Logonomics](https://reader033.vdocuments.net/reader033/viewer/2022051612/54c1bc3d4a795955738b45e0/html5/thumbnails/51.jpg)
Bonus Challenge to All
• Take the Vagrant VM and integrate Logstash shipper with configuraZon files.
• Add Postgres support (Development Only) • Basic syslog funcZonality for CentOs • Custom Log Interface for a B2
![Page 52: Logonomics](https://reader033.vdocuments.net/reader033/viewer/2022051612/54c1bc3d4a795955738b45e0/html5/thumbnails/52.jpg)
Let’s Add-‐on to the IniZaZve developer.blackboard.com