centralized logging
DESCRIPTION
A presentation I did on what, why, how, and benefits of centralized logging in the Enterprise. This presentation was focused on implementing centralized logging in a environment that is mostly .NET/Windows.TRANSCRIPT
![Page 1: Centralized logging](https://reader035.vdocuments.net/reader035/viewer/2022062300/558789dbd8b42a42318b46c9/html5/thumbnails/1.jpg)
Centralized Logging
![Page 2: Centralized logging](https://reader035.vdocuments.net/reader035/viewer/2022062300/558789dbd8b42a42318b46c9/html5/thumbnails/2.jpg)
Logging provides value.
How’s that now?
![Page 3: Centralized logging](https://reader035.vdocuments.net/reader035/viewer/2022062300/558789dbd8b42a42318b46c9/html5/thumbnails/3.jpg)
How does logging provide value?
We can gather information on the system while it’s in operation. We can use this information to provide more value to the business.
![Page 4: Centralized logging](https://reader035.vdocuments.net/reader035/viewer/2022062300/558789dbd8b42a42318b46c9/html5/thumbnails/4.jpg)
More Value?Information gathered from logs allows you to proactively take actions that will keep systems providing the most value to the business.
![Page 5: Centralized logging](https://reader035.vdocuments.net/reader035/viewer/2022062300/558789dbd8b42a42318b46c9/html5/thumbnails/5.jpg)
Talk in circles much?
Centralized logging provides visibility into the operational efficiency of the system and process.
![Page 6: Centralized logging](https://reader035.vdocuments.net/reader035/viewer/2022062300/558789dbd8b42a42318b46c9/html5/thumbnails/6.jpg)
A story on operational sleep efficiency
![Page 7: Centralized logging](https://reader035.vdocuments.net/reader035/viewer/2022062300/558789dbd8b42a42318b46c9/html5/thumbnails/7.jpg)
How is it done?
Logs for most/all systems are shipped, parsed and stored in a central location for monitoring and decision making.
![Page 8: Centralized logging](https://reader035.vdocuments.net/reader035/viewer/2022062300/558789dbd8b42a42318b46c9/html5/thumbnails/8.jpg)
Um…sounds like a bunch of work.
IIS LogsSQL LogsRouter LogsEtc.
![Page 9: Centralized logging](https://reader035.vdocuments.net/reader035/viewer/2022062300/558789dbd8b42a42318b46c9/html5/thumbnails/9.jpg)
Initial focus is on application logging for in-house applications.
![Page 10: Centralized logging](https://reader035.vdocuments.net/reader035/viewer/2022062300/558789dbd8b42a42318b46c9/html5/thumbnails/10.jpg)
Where we are…
Where many are…we’re not alone.
System logs file to directory
System logs file to database
?System
logs file to directory
![Page 11: Centralized logging](https://reader035.vdocuments.net/reader035/viewer/2022062300/558789dbd8b42a42318b46c9/html5/thumbnails/11.jpg)
Where we want to go.
A system that has the ability to store and analyze log data from any system that provides it.
Shipper
App
ViewerParsing, Indexing,
Storage
Logger
Systems
![Page 12: Centralized logging](https://reader035.vdocuments.net/reader035/viewer/2022062300/558789dbd8b42a42318b46c9/html5/thumbnails/12.jpg)
Logging Issuesto resolve
![Page 13: Centralized logging](https://reader035.vdocuments.net/reader035/viewer/2022062300/558789dbd8b42a42318b46c9/html5/thumbnails/13.jpg)
Access To LogGetting access to production servers is not always easy.
![Page 14: Centralized logging](https://reader035.vdocuments.net/reader035/viewer/2022062300/558789dbd8b42a42318b46c9/html5/thumbnails/14.jpg)
Standardizing Logging format
While enabling extensibility…
Log(“Error Occurred, 45, 64”)
Log({Error : “Doh!”,Exeception : ex})
Mobile Client Log
Service Log
Database Log
MVC Application LogDNN Log
Web Client(js) Log
![Page 15: Centralized logging](https://reader035.vdocuments.net/reader035/viewer/2022062300/558789dbd8b42a42318b46c9/html5/thumbnails/15.jpg)
Access to all log data
DB
Service/Logic
Data Tier
Web ClientMobile Client
Logs
Logs
Logs
Logs
Logs
![Page 16: Centralized logging](https://reader035.vdocuments.net/reader035/viewer/2022062300/558789dbd8b42a42318b46c9/html5/thumbnails/16.jpg)
Use log data for…
• Troubleshoot errors and design• Monitoring and notification• Auditing
![Page 17: Centralized logging](https://reader035.vdocuments.net/reader035/viewer/2022062300/558789dbd8b42a42318b46c9/html5/thumbnails/17.jpg)
Business decisions (Analytics)
Upcoming version of Kibana!
![Page 18: Centralized logging](https://reader035.vdocuments.net/reader035/viewer/2022062300/558789dbd8b42a42318b46c9/html5/thumbnails/18.jpg)
The Good
& Bad
of existing solutions
![Page 19: Centralized logging](https://reader035.vdocuments.net/reader035/viewer/2022062300/558789dbd8b42a42318b46c9/html5/thumbnails/19.jpg)
Windows Event Log
• Low Cost – Built in.• Built in UI• Can forward logs to a central server
Good
![Page 20: Centralized logging](https://reader035.vdocuments.net/reader035/viewer/2022062300/558789dbd8b42a42318b46c9/html5/thumbnails/20.jpg)
Windows Event Log
• Event entries stored in memory.• Have to configure logging per server.• Service interface for other clients to
hook into not available.
Bad
![Page 21: Centralized logging](https://reader035.vdocuments.net/reader035/viewer/2022062300/558789dbd8b42a42318b46c9/html5/thumbnails/21.jpg)
Log to central SQL database
• Low Cost – SQL in house, create an database/table.
Good
![Page 22: Centralized logging](https://reader035.vdocuments.net/reader035/viewer/2022062300/558789dbd8b42a42318b46c9/html5/thumbnails/22.jpg)
Log to central SQL database
• Relational (Schema based) format challenging to use for unstructured data.
• logging straight to database could degrade application performance.
Bad
![Page 23: Centralized logging](https://reader035.vdocuments.net/reader035/viewer/2022062300/558789dbd8b42a42318b46c9/html5/thumbnails/23.jpg)
Commercial Hosted Tools
• Handles multiple log formats• Robust charting/searching Features
Good
![Page 24: Centralized logging](https://reader035.vdocuments.net/reader035/viewer/2022062300/558789dbd8b42a42318b46c9/html5/thumbnails/24.jpg)
• Turned down previously• $$$ - pricey for amount of
space allotted
Bad
Commercial Hosted Tools
![Page 25: Centralized logging](https://reader035.vdocuments.net/reader035/viewer/2022062300/558789dbd8b42a42318b46c9/html5/thumbnails/25.jpg)
And the winner is….
A Hybrid Approach.
![Page 26: Centralized logging](https://reader035.vdocuments.net/reader035/viewer/2022062300/558789dbd8b42a42318b46c9/html5/thumbnails/26.jpg)
Application
Redis Queue
Log Indexer
Kibana
Application
EL
Application
Elastic Search
Log
Lo
g
Log
ELEL
Rest APIArchival
![Page 27: Centralized logging](https://reader035.vdocuments.net/reader035/viewer/2022062300/558789dbd8b42a42318b46c9/html5/thumbnails/27.jpg)
Low(No) Cost
OSS FTW!
![Page 28: Centralized logging](https://reader035.vdocuments.net/reader035/viewer/2022062300/558789dbd8b42a42318b46c9/html5/thumbnails/28.jpg)
Extensible
Using a Pub/Sub pattern from the queue will allow other components to subscribe to it.
Redis Queue
Log Indexer(Publisher)
Statistics(Subscriber)
Notifications(Subscriber)
![Page 29: Centralized logging](https://reader035.vdocuments.net/reader035/viewer/2022062300/558789dbd8b42a42318b46c9/html5/thumbnails/29.jpg)
Runs on Windows
Java/Ruby based tools run on windows with some setup.
![Page 30: Centralized logging](https://reader035.vdocuments.net/reader035/viewer/2022062300/558789dbd8b42a42318b46c9/html5/thumbnails/30.jpg)
Blocking
Asynchronous
Mes
sage
Queu
e
No
![Page 31: Centralized logging](https://reader035.vdocuments.net/reader035/viewer/2022062300/558789dbd8b42a42318b46c9/html5/thumbnails/31.jpg)
Why ElasticSearch?
You mean aside from the fact that StackOverflow founder uses it?
![Page 32: Centralized logging](https://reader035.vdocuments.net/reader035/viewer/2022062300/558789dbd8b42a42318b46c9/html5/thumbnails/32.jpg)
Inverted Index“In computer science, an inverted index (also referred to as postings file or inverted file) is anindex data structure storing a mapping from content, such as words or numbers, to its locations in adatabase file, or in a document or a set of documents. The purpose of an inverted index is to allow fast full text searches, at a cost of
increased processing when a document is added to the database.” via http://en.wikipedia.org/wiki/Inverted_index
Page
WordWordWord
Word
PagePagePage
Regular Index Inverted Index
![Page 33: Centralized logging](https://reader035.vdocuments.net/reader035/viewer/2022062300/558789dbd8b42a42318b46c9/html5/thumbnails/33.jpg)
Documented Oriented• No need to define schema upfront• Can store, index and quickly search unstructured data.• Schema can be defined per type for customization of
the indexing process.
![Page 34: Centralized logging](https://reader035.vdocuments.net/reader035/viewer/2022062300/558789dbd8b42a42318b46c9/html5/thumbnails/34.jpg)
Big Community
• VIA http://www.elasticsearch.org/community/
![Page 35: Centralized logging](https://reader035.vdocuments.net/reader035/viewer/2022062300/558789dbd8b42a42318b46c9/html5/thumbnails/35.jpg)
Easy to scale
• Distributed by nature.• Indexes broken down by shards with 0 or more
replicas.
![Page 36: Centralized logging](https://reader035.vdocuments.net/reader035/viewer/2022062300/558789dbd8b42a42318b46c9/html5/thumbnails/36.jpg)
Easy Level 1 Rest API
• Well documented and straight forward api, which makes it easy to build a client for it.
![Page 37: Centralized logging](https://reader035.vdocuments.net/reader035/viewer/2022062300/558789dbd8b42a42318b46c9/html5/thumbnails/37.jpg)
What’s Needed recap• Server for Elastic Search (Windows/Linux)• Redis Server• Windows service to index logs from queue.• Modify Enterprise Library Logger to send to
queue.
![Page 38: Centralized logging](https://reader035.vdocuments.net/reader035/viewer/2022062300/558789dbd8b42a42318b46c9/html5/thumbnails/38.jpg)
In closingGathering the data is one part. Logging enough information and knowing what questions your are looking to answer is another part.
![Page 39: Centralized logging](https://reader035.vdocuments.net/reader035/viewer/2022062300/558789dbd8b42a42318b46c9/html5/thumbnails/39.jpg)
References