from&tool&to&team& member:&...
TRANSCRIPT
Copyright © 2014 Splunk Inc.
George Starcher Security Engineer, Peak Hos6ng
From Tool to Team Member: Controlling Systems with Splunk Alert Scripts
About Me ! George Starcher, Informa6on Security Engineer ‒ CISSP, Splunk Cer6fied Knowledge Manager and Splunk
Cer6fied Administrator ! Splunk IRC Channel ! Looking to kick off a Nashville, TN – Splunk User Group • www.georgestarcher.com • www.peakhos6ng.com
2
Agenda
! Splunk from Tool to a Team Member ! How it Works ! GeQng into the Code ‒ Alert Script to Intrusion Preven6on System Control ‒ Alert Script to X-‐ARF Abuse Repor6ng
3
“Using Alert Scripts to take ac6on on our behalf, we can transform Splunk from a tool to a team member.”
Splunk from Tool to Team Member ! Manual Abuse Scanning Process ‒ Reviewed SSH, RDP, VNC etc daily ‒ Consumed 30-‐45 minutes per day ‒ Permanent blacklist entries
! Moved to automated process ‒ Scheduled Splunk Searches driven by any log source ‒ Greatly reduced 6me and sta6c blacklist maintenance ‒ Web Services (REST) calls to the IPS
4
Splunk from Tool to Team Member
5
Outlook Web Access – Phishers
Started Feb 10, 2014
• Blocked for any access from Nigeria every 5 minutes
Expanded Mul6 Country Feb 15, 2014
• Blocked for combina6on from certain countries & a lookup table of hosted providers
Feb 17, 2014
• No6ced unexpected Exchange OWA from Nigeria
Splunk from Tool to Team Member
6
Outlook Web Access – Phishers
Single User by src_ip_country:
Hosted Lookup users by src_ip:
How it Works
How it Works
8
Intrusion Preven6on Appliance
How it Works
9
How it Works
10
h8p://blogs.splunk.com/2011/03/15/storing-‐encrypted-‐credenCals/
h8p://www.georgestarcher.com/splunk-‐alert-‐scripts-‐automaCng-‐control/
How it Works
11
! Setup a service account to own the Alert Searches: svc-‐alert ! Create a role just for the alert account ! That role must have ‘admin_all_objects’ ! The role must have access to all indexes that might have the data for the scheduled search alert
Alert Service Account
How it Works
12
Alert Script in Ac6on
13
! Avoids manual repor6ng ! Ensures 6mely ac6on ! Consistent Repor6ng Format ! Accurate Evidence Data ! Works around the clock and doesn't need coffee
X-‐ARF Abuse Repor6ng
Alert Script in Ac6on
14
X-‐ARF Abuse Repor6ng
Alert Script in Ac6on
15
X-‐ARF Abuse Repor6ng
Crawling into the Code
@SplunkDev Team -‐ THANKS!!
17
@gblock -‐ Glenn Block
@damiendallimore -‐Damien Dallimore
David Noble -‐ Twiner App
Where Can You Get The Code?
18
! Github Repository ‒ hnps://github.com/georgestarcher/Splunk-‐Alert ‒ General Intrusion Preven6on System Example Code ‒ Google Spreadsheet Upload Code ‒ X-‐ARF Abuse Repor6ng Code
! The Google Spreadsheet Example ‒ hnp://www.georgestarcher.com/splunk-‐alert-‐scripts-‐automa6ng-‐control/
Arguments Sent to Alert Scripts
19
h8p://docs.splunk.com/DocumentaCon/Splunk/6.1.3/Alert/Configuringscriptedalerts
! SPLUNK_ARG_0 Script name ! SPLUNK_ARG_1 Number of events returned ! SPLUNK_ARG_2 Search terms ! SPLUNK_ARG_3 Fully qualified query string ! SPLUNK_ARG_4 Name of report ! SPLUNK_ARG_5 Trigger reason (for example, "The number of events was greater than 1")
! SPLUNK_ARG_6 Browser URL to view the report ! SPLUNK_ARG_8 File in which the results for this search are stored (contains raw results)
The Code Modules – IPS
20
! creden6alsFromSplunk.py ‒ A Python class to fetch the saved service account
! targetlist.py ‒ The Python class for data to be handled
! ips.py ‒ The Python class for an IPS rest API interface
! alert_script.py ‒ The main Python alert script
creden6alsFromSplunk.py
21
! A re-‐usable Python class to fetch stored user creden6als from Splunk ! Provide the app where creden6aled is stored: splunkapp ! Provide the purpose name used when saving the creden6als: realm ! Provide the username to be retrieved: username ! Call the getPassword method
creden6alsFromSplunk.py
22
# Define the source in Splunk for the stored credenCal splunkapp = "myadmin" realm = 'ips' username = 'splunk'
# Get the stored credenCal from Splunk try: ipsCredenCal.getPassword(sessionKey) except ExcepCon, e: logError("Splunk CredenCal Error: %s" % str(e)) exitAlertScript(_SYS_EXIT_FAILED_SPLUNK_AUTH)
# Define the ips connecCon ipsCredenCal = credenCal(splunkapp,realm,username)
from alert_script.py
targetlist.py
23
! A simple Python class for a single column list of source IPs ! Populated by the alert search returning only source IPs ! Takes argument of path to the search results to load the list
# Obtain the path to the alert events compressed file and load the search results to the list alertEventsFile = os.environ['SPLUNK_ARG_8']
try: alertTargetList = targetlist(alertEventsFile) except ExcepCon, e: logError("Target File Error: %s" % str(e)) exitAlertScript(_SYS_EXIT_FAILED_TARGET_FILE)
from alert_script.py
ips.py
24
! An example Python class to interface with our Intrusion Detec6on System Rest API
! Setup and retrieve the creden6al from splunk: ipsCredenCal ! Provide the IPS quaran6ne policy name: policy_name ! Provide IP address of the IPS management Interface: ips_ip ! Ac6vate the IPS rest connec6on object ! Loop through the alertTargetList having the IPS quaran6ne each IP
Make your Own REST API wrapper class to control other systems
ips.py
25
# AcCve the ips connecCon object try: ssh_ips = ips(ips_ip,ipsCredenCal.username,
ipsCredenCal.password,policy_name) except ExcepCon, e: logError("IPS Error: %s" % str(e)) exitAlertScript(_SYS_EXIT_FAILED_IPS)
from alert_script.py
alert_script.py
26
! The main script called by Splunk for our alert search ! Imports all our classes ! Parses the sessionKey ! Connects to our IPS ! Pulls in the search result list of IP addresses ! Loops through the IP list and tells the IPS to quaran6ne them
alert_script.py
27
The Hash Bang: #!/opt/splunk/bin/python
# QuaranCne each source ip in the alert results table
for address in alertTargetList.targetlist: try: ssh_ips.addQuaranCne(address) except ExcepCon, e: logError("IPS QuaranCne Error: %s" % str(e)) exitAlertScript(_SYS_EXIT_FAILED_IPS)
# Obtain the Splunk authenCcaCon session key … # Adjust the returned sessionKey text based on Splunk version …
Extended Abuse Repor6ng -‐ X-‐ARF
28
! Much more complex code ! Search results driving results is a table of data not a simple IP list ! Pulls email seQngs from Splunk ! Builds the email body using the Python Mako template (mail merge to search results)
! Improved alert script ac6on logging sending into index=_internal ! Anaches Alert Event Search results from Splunk REST API Calls
h8p://www.x-‐arf.org/
BONUS
The Code Modules -‐ X-‐ARF
29
! abuselist.py ‒ The data to be handled
! emailSplunkXARF.py ‒ A python class to fetch the saved service account
! xarf-‐abuse.tmpl ‒ Abuse report Email mako template
! alert_to_xarf.py ‒ The main alert script
abuselist.py
30
! Method getEvidence holds the evidence search executed against the Splunk REST API
! This method also manipulates the earliest/latest 6mestamp coming from the search results automa6cally to go into the detail evidence search
emailSplunkXARF.py ! Method getMailSeQngs is Splunk REST API call to fetch the seQngs from your Splunk server
alert_to_xarf.py
31
! All the X-‐ARF values are at the top of the script ! Method getSplunkVersion gets the running Splunk version from the REST API to help auto adjust the sessionKey
! Method getSplunkUser gets the username the Alert executed under from Splunk needed for the evidence search fetch
! Logging writes with proper 6mestamp GMT to $SPLUNK_HOME/var/log/splunk/…
You could use this to make your own highly customized alert email based on search results
Thank You!
32
Other resources Splunk IRC ( EFNet #splunk ) Splunk Answers ( hnp://answers.splunk.com ) Splunk community wiki ( hnp://wiki.splunk.com ) hnp://www.georgestarcher.com/ hnp://blog.splunk.com/ hnp://www.meetup.com/Splunk/Nashville-‐TN/
Other “must-‐see” .conf 2014 presentaCons ! Avoid the SSLippery Slope of Default SSL -‐ Duane Waddle and George Starcher ! In Depth With Deployment Server -‐ Dave Shpritz, Aplura ! Using Lesser Known Commands in Splunk Search Processing Language (SPL) -‐ Kyle Smith, The Hershey Company
! Masters of IRC -‐ panel talk on the Splunk Community Stage
THANK YOU