log analysis with - nathan hunstad
TRANSCRIPT
![Page 1: Log Analysis with - Nathan Hunstad](https://reader031.vdocuments.net/reader031/viewer/2022020702/61f9b3c979eb8c4e1e6a11b6/html5/thumbnails/1.jpg)
Log AnalysisLog Analysiswithwith
Presenter: Nathan HunstadMay 2015
![Page 2: Log Analysis with - Nathan Hunstad](https://reader031.vdocuments.net/reader031/viewer/2022020702/61f9b3c979eb8c4e1e6a11b6/html5/thumbnails/2.jpg)
Obligatory Disclaimer
● This talk represents my own work: I am not representing any employer, organization, or affiliated group, past, present, or future
● This talk is based on my experiences in my home lab network and not in an enterprise setting
● This is an overview only and is provided without warranty: do not rely on what you learn here for compliance or legal obligations!
![Page 3: Log Analysis with - Nathan Hunstad](https://reader031.vdocuments.net/reader031/viewer/2022020702/61f9b3c979eb8c4e1e6a11b6/html5/thumbnails/3.jpg)
What is Log Analysis?
Not this:
![Page 4: Log Analysis with - Nathan Hunstad](https://reader031.vdocuments.net/reader031/viewer/2022020702/61f9b3c979eb8c4e1e6a11b6/html5/thumbnails/4.jpg)
What is Log Analysis?
Or this:
![Page 5: Log Analysis with - Nathan Hunstad](https://reader031.vdocuments.net/reader031/viewer/2022020702/61f9b3c979eb8c4e1e6a11b6/html5/thumbnails/5.jpg)
What is Log Analysis, really?
● Forensics: Reconstructing events that have already happened
● Incident Response: Acting on logs in real-time to identify, contain, and remediate security incidents
● Troubleshooting: Evaluating systems for faults or unintended behavior and fixing as necessary
![Page 6: Log Analysis with - Nathan Hunstad](https://reader031.vdocuments.net/reader031/viewer/2022020702/61f9b3c979eb8c4e1e6a11b6/html5/thumbnails/6.jpg)
Handling Logs
● Help!
![Page 7: Log Analysis with - Nathan Hunstad](https://reader031.vdocuments.net/reader031/viewer/2022020702/61f9b3c979eb8c4e1e6a11b6/html5/thumbnails/7.jpg)
Splunk
● Splunk captures all kinds of machine data – app log files, syslog, text files, configuration files...basically any text data can be ingested
● Splunk provides a powerful search engine based on MapReduce for fast searching1
● Splunk has add-ins that allow for quickly setting up dashboards and reports for common log sources
● No, I do not work for Splunk
1 https://www.splunk.com/content/dam/splunk2/pdfs/technical-briefs/splunk-and-mapreduce.pdf
![Page 8: Log Analysis with - Nathan Hunstad](https://reader031.vdocuments.net/reader031/viewer/2022020702/61f9b3c979eb8c4e1e6a11b6/html5/thumbnails/8.jpg)
Splunk Licensing
● Splunk Enterprise: based on log volume
● Splunk Free: fewer features, 500MB/day
– Go over? You will lose search access!– But good enough for home use
![Page 9: Log Analysis with - Nathan Hunstad](https://reader031.vdocuments.net/reader031/viewer/2022020702/61f9b3c979eb8c4e1e6a11b6/html5/thumbnails/9.jpg)
Splunk Licensing
● Average Logging Volume
![Page 10: Log Analysis with - Nathan Hunstad](https://reader031.vdocuments.net/reader031/viewer/2022020702/61f9b3c979eb8c4e1e6a11b6/html5/thumbnails/10.jpg)
Adding Data to Splunk
![Page 11: Log Analysis with - Nathan Hunstad](https://reader031.vdocuments.net/reader031/viewer/2022020702/61f9b3c979eb8c4e1e6a11b6/html5/thumbnails/11.jpg)
Getting Data Into Splunk
● Splunk Forwarder
– Install on any system to read log files locally and forward to Splunk Indexer
– Versions available for Windows, MacOS, Linux, Solaris, HPUX, AIX, and FreeBSD
– Configure using GUI or edit .conf files
![Page 12: Log Analysis with - Nathan Hunstad](https://reader031.vdocuments.net/reader031/viewer/2022020702/61f9b3c979eb8c4e1e6a11b6/html5/thumbnails/12.jpg)
Getting Data Into Splunk
● Listen on port
– Splunk daemon binds to a port to listen for traffic (TCP or UDP)
– Typically used with syslog data
![Page 13: Log Analysis with - Nathan Hunstad](https://reader031.vdocuments.net/reader031/viewer/2022020702/61f9b3c979eb8c4e1e6a11b6/html5/thumbnails/13.jpg)
Getting Data Into Splunk
● Monitor Files/Directories
– Splunk daemon monitors individual files or an entire directory for new files/changes to files
– Computes CRC and bytes read on files to detect changes
– Can automatically decompress common formats like zip files
![Page 14: Log Analysis with - Nathan Hunstad](https://reader031.vdocuments.net/reader031/viewer/2022020702/61f9b3c979eb8c4e1e6a11b6/html5/thumbnails/14.jpg)
Getting Data Into Splunk
● Remote Hosts
– What if you can't install a forwarder on a remote host (for example, your shared web host?)
– My Solution: cron job + monitoring files
![Page 15: Log Analysis with - Nathan Hunstad](https://reader031.vdocuments.net/reader031/viewer/2022020702/61f9b3c979eb8c4e1e6a11b6/html5/thumbnails/15.jpg)
Splunk Basics
![Page 16: Log Analysis with - Nathan Hunstad](https://reader031.vdocuments.net/reader031/viewer/2022020702/61f9b3c979eb8c4e1e6a11b6/html5/thumbnails/16.jpg)
Indexes
● Indexes are the logical buckets into which data is stored● By default, all data gets stored in the main index, but
other indexes can be defined● A number of internal indexes exist for tracking Splunk
functionality and start with _, such as _internal and _audit
● Data retention and access control* is done on a per-index basis
![Page 17: Log Analysis with - Nathan Hunstad](https://reader031.vdocuments.net/reader031/viewer/2022020702/61f9b3c979eb8c4e1e6a11b6/html5/thumbnails/17.jpg)
Buckets
● Buckets are collections of index data and metadata
● Buckets age through several stages: Hot, Warm, Cold, and Frozen
● Not terribly important for home use, but managing retention becomes important for large data sets
![Page 18: Log Analysis with - Nathan Hunstad](https://reader031.vdocuments.net/reader031/viewer/2022020702/61f9b3c979eb8c4e1e6a11b6/html5/thumbnails/18.jpg)
My Environment
● Splunk server: located on server running CentOS
● Feeds from VMs, Windows desktops, EdgeOS router, managed switch, application logs, external website
![Page 19: Log Analysis with - Nathan Hunstad](https://reader031.vdocuments.net/reader031/viewer/2022020702/61f9b3c979eb8c4e1e6a11b6/html5/thumbnails/19.jpg)
My Environment
● Data is split up into multiple indexes for logical grouping
● Indexes for firewall, switch, Linux, Windows, website, and BOINC events, plus a throwaway index for testing
![Page 20: Log Analysis with - Nathan Hunstad](https://reader031.vdocuments.net/reader031/viewer/2022020702/61f9b3c979eb8c4e1e6a11b6/html5/thumbnails/20.jpg)
Windows Events
● Events from Security, Application, and System logs
![Page 21: Log Analysis with - Nathan Hunstad](https://reader031.vdocuments.net/reader031/viewer/2022020702/61f9b3c979eb8c4e1e6a11b6/html5/thumbnails/21.jpg)
Windows Events
● PerfMon performance monitoring events
![Page 22: Log Analysis with - Nathan Hunstad](https://reader031.vdocuments.net/reader031/viewer/2022020702/61f9b3c979eb8c4e1e6a11b6/html5/thumbnails/22.jpg)
Linux Events
● Syslog events
![Page 23: Log Analysis with - Nathan Hunstad](https://reader031.vdocuments.net/reader031/viewer/2022020702/61f9b3c979eb8c4e1e6a11b6/html5/thumbnails/23.jpg)
Website Logs
● Multiple access logs
![Page 24: Log Analysis with - Nathan Hunstad](https://reader031.vdocuments.net/reader031/viewer/2022020702/61f9b3c979eb8c4e1e6a11b6/html5/thumbnails/24.jpg)
Website Logs
● Apache access_combined
![Page 25: Log Analysis with - Nathan Hunstad](https://reader031.vdocuments.net/reader031/viewer/2022020702/61f9b3c979eb8c4e1e6a11b6/html5/thumbnails/25.jpg)
Firewall Logs
● Dropped and specific accepted connections
![Page 26: Log Analysis with - Nathan Hunstad](https://reader031.vdocuments.net/reader031/viewer/2022020702/61f9b3c979eb8c4e1e6a11b6/html5/thumbnails/26.jpg)
Switch Logs
● Connected devices
![Page 27: Log Analysis with - Nathan Hunstad](https://reader031.vdocuments.net/reader031/viewer/2022020702/61f9b3c979eb8c4e1e6a11b6/html5/thumbnails/27.jpg)
Application Logs
● BOINC (Berkeley Open Infrastructure for Network Computing) events
![Page 28: Log Analysis with - Nathan Hunstad](https://reader031.vdocuments.net/reader031/viewer/2022020702/61f9b3c979eb8c4e1e6a11b6/html5/thumbnails/28.jpg)
Basic Search Syntax
![Page 29: Log Analysis with - Nathan Hunstad](https://reader031.vdocuments.net/reader031/viewer/2022020702/61f9b3c979eb8c4e1e6a11b6/html5/thumbnails/29.jpg)
Search Syntax
● Basic search: just type in what you want to see
![Page 30: Log Analysis with - Nathan Hunstad](https://reader031.vdocuments.net/reader031/viewer/2022020702/61f9b3c979eb8c4e1e6a11b6/html5/thumbnails/30.jpg)
Search Syntax
● Limiting by fields
![Page 31: Log Analysis with - Nathan Hunstad](https://reader031.vdocuments.net/reader031/viewer/2022020702/61f9b3c979eb8c4e1e6a11b6/html5/thumbnails/31.jpg)
Search Syntax
● Counting events: stats count
![Page 32: Log Analysis with - Nathan Hunstad](https://reader031.vdocuments.net/reader031/viewer/2022020702/61f9b3c979eb8c4e1e6a11b6/html5/thumbnails/32.jpg)
Search Syntax
● Top events: top
![Page 33: Log Analysis with - Nathan Hunstad](https://reader031.vdocuments.net/reader031/viewer/2022020702/61f9b3c979eb8c4e1e6a11b6/html5/thumbnails/33.jpg)
Search Syntax
● Bucketing events and charting: timechart
![Page 34: Log Analysis with - Nathan Hunstad](https://reader031.vdocuments.net/reader031/viewer/2022020702/61f9b3c979eb8c4e1e6a11b6/html5/thumbnails/34.jpg)
Security Events
![Page 35: Log Analysis with - Nathan Hunstad](https://reader031.vdocuments.net/reader031/viewer/2022020702/61f9b3c979eb8c4e1e6a11b6/html5/thumbnails/35.jpg)
Brute Force Windows
● Using ncrack against RDP
![Page 36: Log Analysis with - Nathan Hunstad](https://reader031.vdocuments.net/reader031/viewer/2022020702/61f9b3c979eb8c4e1e6a11b6/html5/thumbnails/36.jpg)
Brute Force Windows
● Success!
![Page 37: Log Analysis with - Nathan Hunstad](https://reader031.vdocuments.net/reader031/viewer/2022020702/61f9b3c979eb8c4e1e6a11b6/html5/thumbnails/37.jpg)
Brute Force Linux
● Using Metasploitable ssh_login module and default root_userpass.txt
![Page 38: Log Analysis with - Nathan Hunstad](https://reader031.vdocuments.net/reader031/viewer/2022020702/61f9b3c979eb8c4e1e6a11b6/html5/thumbnails/38.jpg)
Port Scanning (External)
● Port Scanning: Same source IP, multiple destination ports
![Page 39: Log Analysis with - Nathan Hunstad](https://reader031.vdocuments.net/reader031/viewer/2022020702/61f9b3c979eb8c4e1e6a11b6/html5/thumbnails/39.jpg)
Port Scanning (Internal)
● Port Scanning: Same source IP, multiple destination ports
![Page 40: Log Analysis with - Nathan Hunstad](https://reader031.vdocuments.net/reader031/viewer/2022020702/61f9b3c979eb8c4e1e6a11b6/html5/thumbnails/40.jpg)
SQL Injection
● sqlmap against DVWA
● Apache logs sent to Splunk
![Page 41: Log Analysis with - Nathan Hunstad](https://reader031.vdocuments.net/reader031/viewer/2022020702/61f9b3c979eb8c4e1e6a11b6/html5/thumbnails/41.jpg)
Blind SQL Injection
● sqlmap/DVWA
![Page 42: Log Analysis with - Nathan Hunstad](https://reader031.vdocuments.net/reader031/viewer/2022020702/61f9b3c979eb8c4e1e6a11b6/html5/thumbnails/42.jpg)
XSS
● Persistent XSS on DVWA
![Page 43: Log Analysis with - Nathan Hunstad](https://reader031.vdocuments.net/reader031/viewer/2022020702/61f9b3c979eb8c4e1e6a11b6/html5/thumbnails/43.jpg)
Mimikatz
● Running mimikatz to dump hashes
● Nothing happens
![Page 44: Log Analysis with - Nathan Hunstad](https://reader031.vdocuments.net/reader031/viewer/2022020702/61f9b3c979eb8c4e1e6a11b6/html5/thumbnails/44.jpg)
Correlation
● Transactions: group events together that match a pattern
● Successful login following failed logins
![Page 45: Log Analysis with - Nathan Hunstad](https://reader031.vdocuments.net/reader031/viewer/2022020702/61f9b3c979eb8c4e1e6a11b6/html5/thumbnails/45.jpg)
Correlation
● Show attackers in a table
● index=linux | transaction host,rhost startswith="eventtype=sshd-login-failure" endswith="eventtype=ssh_open" | bucket _time span=30m | table _time, rhost
![Page 46: Log Analysis with - Nathan Hunstad](https://reader031.vdocuments.net/reader031/viewer/2022020702/61f9b3c979eb8c4e1e6a11b6/html5/thumbnails/46.jpg)
More Splunking
![Page 47: Log Analysis with - Nathan Hunstad](https://reader031.vdocuments.net/reader031/viewer/2022020702/61f9b3c979eb8c4e1e6a11b6/html5/thumbnails/47.jpg)
Field Extraction
● Splunk can handle some log types automatically pretty well, but adding rules for field extraction can help with searching and indexing
● A number of extractions come with Splunk ready for use, or you can add your own
● Uses regex for extraction
![Page 48: Log Analysis with - Nathan Hunstad](https://reader031.vdocuments.net/reader031/viewer/2022020702/61f9b3c979eb8c4e1e6a11b6/html5/thumbnails/48.jpg)
Lookups
● Uploading CSV files for extracting or expanding on data in logs
![Page 49: Log Analysis with - Nathan Hunstad](https://reader031.vdocuments.net/reader031/viewer/2022020702/61f9b3c979eb8c4e1e6a11b6/html5/thumbnails/49.jpg)
Lookups
● Previous timechart, now with names
![Page 50: Log Analysis with - Nathan Hunstad](https://reader031.vdocuments.net/reader031/viewer/2022020702/61f9b3c979eb8c4e1e6a11b6/html5/thumbnails/50.jpg)
Data Models
● Data Models are a powerful way of structuring data to generate specialized searches and visualizations
● Can be used to generate pivot tables and other complex objects
![Page 51: Log Analysis with - Nathan Hunstad](https://reader031.vdocuments.net/reader031/viewer/2022020702/61f9b3c979eb8c4e1e6a11b6/html5/thumbnails/51.jpg)
Pivot Tables
● Based on defined data models
● Display data in tabular format
![Page 52: Log Analysis with - Nathan Hunstad](https://reader031.vdocuments.net/reader031/viewer/2022020702/61f9b3c979eb8c4e1e6a11b6/html5/thumbnails/52.jpg)
Dashboards
● Bringing all your data to one spot, with user-selectable attributes
![Page 53: Log Analysis with - Nathan Hunstad](https://reader031.vdocuments.net/reader031/viewer/2022020702/61f9b3c979eb8c4e1e6a11b6/html5/thumbnails/53.jpg)
Visualizations
● Looking closer at the “Website Attacks” dashboard:
– Logarithmic Y-axis
– Daily Buckets
![Page 54: Log Analysis with - Nathan Hunstad](https://reader031.vdocuments.net/reader031/viewer/2022020702/61f9b3c979eb8c4e1e6a11b6/html5/thumbnails/54.jpg)
Visualizations
● Grouping Events
– Attacks: index=website joomlafailure sourcetype="php_error" | transaction IP maxpause=1h maxevents=5000| where eventcount>1 | table _time, IP, eventcount
– Port Scanning: index=firewall RuleName=WAN-*default-D | bucket _time span=30 | eventstats dc(DPT) AS PortsScanned by SRC, _time | where PortsScanned > 5 | dedup SRC, PortsScanned | table SRC, PortsScanned, _time
![Page 55: Log Analysis with - Nathan Hunstad](https://reader031.vdocuments.net/reader031/viewer/2022020702/61f9b3c979eb8c4e1e6a11b6/html5/thumbnails/55.jpg)
Visualizations
● Firewall Drops
![Page 56: Log Analysis with - Nathan Hunstad](https://reader031.vdocuments.net/reader031/viewer/2022020702/61f9b3c979eb8c4e1e6a11b6/html5/thumbnails/56.jpg)
Geolocation
● Splunk can Geolocate IP addresses
![Page 57: Log Analysis with - Nathan Hunstad](https://reader031.vdocuments.net/reader031/viewer/2022020702/61f9b3c979eb8c4e1e6a11b6/html5/thumbnails/57.jpg)
Geolocation
● Search: index=website joomlafailure sourcetype="php_error" | transaction IP maxpause=1h maxevents=5000 | where eventcount>1 | iplocation IP | geostats latfield=lat longfield=lon sum(eventcount)
![Page 58: Log Analysis with - Nathan Hunstad](https://reader031.vdocuments.net/reader031/viewer/2022020702/61f9b3c979eb8c4e1e6a11b6/html5/thumbnails/58.jpg)
Splunk TAs
● Splunk comes with TAs (Technology add-on) with pre-defined field extractions, transformations, and dashboards
![Page 59: Log Analysis with - Nathan Hunstad](https://reader031.vdocuments.net/reader031/viewer/2022020702/61f9b3c979eb8c4e1e6a11b6/html5/thumbnails/59.jpg)
Happy Logging!