splunk enterprise security for proactive monitoring · tips, tricks, and analytics. purpose...

Splunk Enterprise SecurityFor Proactive Monitoring

AKA: Enterprise SecurityTips, Tricks, and Analytics

Purpose

● Describe Tips for a Clean Setup of ES

● Provide Tricks “From the Field” in Setup/Mgmt

● Demonstrate Analysis With and Without Training Wheels

Who Am I?

● Sean Wilkerson, Partner/Consultant, Aplura

● Speaker at SANS Log Mgmt Summits

● Splunk Pro Serv Partner Since 2008

Splunk/ES Experience

● 20+ ES Engagements

● Dozens of Different Federal Entities

● Many Commercial Customers

● My 4th .conf

● 5+ Years of Splunk Pro Serv.

● 6+ Years using Splunk

● 12+ Years Of Logs (Shell, Scripts, SIM, Splunk)

● 14+ Years of Network --> Systems --> InfoSec

Who Are You?

● You Know a Handful of Splunk Search Cmds

● You Have Worked With Splunk Conf Files

● You Know Generally What ES is demos/talks

● You May be a Splunk/ES User/Administrator

● You Are Analysts!! <--- Really Important

Content Available Now!

aplura.com/splunkconf2013

Splunk App for Enterprise Security

Scalability to manage multi-terabytes of real-time and historical data

Pre-built security correlation rules, reports, and dashboards

Statistical analysis for defining ‘normal’

Incident investigation and management framework

Solution with out-of-the-box content to manage known and unknown threats.

Security AnalystsSOC Staff Security Execs/Mgrs

Security Auditors

vocab(ES)

● ES = Enterprise Security

● TA = Technology Add-on (fields and tags)

● SA = Security Add-on (searches and corr logic)

● DA = Domain Add-on (dashboards)

● Macros = Shortcut to Splunk search string

● Correlation = Notable Event Searches

● Onboard = Inputs and TAs

● CIM = Common Information Model

Splunk Enterprise Security

Tips

ES From 10,001 Feet

● ES = Well-Organized Deployment

● Good Organization = Free Correlation

● CIM = Babelfish (One Language)

● Allows Fewer and Clearer Correlations

● Intelligently Doing More with Less...Overhead

● Improves the Speed to Root Cause Analysis

Deployment Steps

● First, Solidify Architecture

● Install: DS, SH->IDX; Validate Storage, etc.

● Ensure ES Storage Supports TSIDX (100GB is 340G/yr w/ ES-2.x)

● Onboard at least: Firewall, WEL (AD), IDS, AV

● Start ES and Validate TAs

● Enable and Schedule Desired Correlations

● Integrate Assets and Identities

● Onboard Other Supported Data-Sources

● Onboard Custom Data-Sources

● Tune and Optimize

General Housekeeping

● ASAP Start Defining Assets

● Time-Audit Before It's Too Late

● RT > index=* | eval timeDiff=_indextime-_time | timechart span=10m avg(timeDiff) by sourcetype

Leverage ES Strengths

● ES Reports on Security-related Decisions

● Information is Grouped into Three Domains:

● Access (Logins, Admin Activity)

● Endpoint (Malware, Systems, Time)

● Network (Firewall, IDS, VA, WebProxy)

● Some Data Doesn't Need ES

● ES Assumes a Framework, So Should You

**Hazards Ahead**

The journey is profitable; however,fumbled steps can land you in peril.

Hazard: Underpowered Hardware

● Splunk (like DBs) Can Run on An Old Laptop

● It doesn't mean that it should!!

● Meet or Exceed the “Reference“ “Architecture“

● Don't Skimp on Hardware!!

● Until [ $IOPS >= 1200 ]; do storage++; done;

Hazard: No EventGen in Production

● EventGen Creates Fake Data for DEMOS

● Do Not Enable This in Production!!

● Really? Do I Have to Say This?

● Yes, I do!!

Hazard: App Isolation● “App Isolation” allows apps to play nice

with each other with little to no regard for precedence.

● For testing: Edit any SA-$NAME/medatadata/local.meta to add your custom app

● For permanence: Edit app SplunkEnterpriseSecuritySuite default/inputs.conf

● Be Mindful of App Isolation – It Can Bite Hard

Hazard: Asset Formatting

● Assets provide the context between the data and correlations. VALUE++

● ES-2.2+ - “Asset Expander” - Validate/Format

● > index=_internal source=*lookup_expander.log

● Temporarily Adjust Input to Shorten Test Cycle

Hazard: RealTime Correlations

● Many of the Correlations are “RealTime”

● Switch these to scheduled (generally speaking)

Hazard: Customizations

● In Splunk – Custom = Immortalized

● This is the local vs default thing...

● Customizations Can Effect the Mechanics of ES

● Leverage the Customizations Encouraged in ES but Don't Make Your Own

● Customizations = Difficult Upgrades

● Customize Correlations With Care

● Do Not Customize Views, Assets-fields, or Scripts

Splunk Enterprise Security

Tricks

Tricks: Dynamic Lists

● This May Look Unexciting, but it is What I Get Asked for the Most

● Dynamic Assets/Identities (via SavedSearch) Whenever Possible

● Use SA-ldapsearch for Both, such as this:

| ldapsearch domain=$domain$ search="(&(objectClass=user)(!(objectClass=computer))(!(displayName=SystemMailbox*)))" attrs="cn,userPrincipalName,sAMAccountName,personalTitle,displayName,givenName,sn,suffix,mail,telephoneNumber,mobile,manager,priority,department,category,watchlist,whenCreated,accountExpires" | fields - _* | rename sAMAccountName as identity, personalTitle as prefix, displayName as nick, givenName as first, sn as last, mail as email, telephoneNumber as phone, mobile as phone2, manager as managedBy, department as bunit, whenCreated as startDate, accountExpires as endDate | table identity, prefix, nick, first, last, suffix, email, phone, phone2, managedBy, priority, bunit, category, watchlist, startDate, endDate | outputlookup simple_identity_lookup

Tricks: Nice Assets

● Create Asset Categories In SavedSearch

| `assets` | mvexpand category | dedup category | sort category | table category | outputlookup category_lookup

● Use CIDR Blocks in Assets

● This Allows for Inclusion/Exclusion of Network by category reference. This is big!

– E.G. All IDS alerts by category=oracle_cluster

Tricks: Nice Assets 2

● Plan Asset Categories and Benefit

● Use Built-ins When Available (e.g. email_servers)

● Don't Make More Granularity Than You Can Use

● Plan Supportive Naming Scheme For CIDR...

– foonet_nyc_dmz

– foonet_nyc_users

– foonet_chg_dmz

Note: Critical Point

Tricks: Upgrade !SNAFU

● Read ReleaseNotes

● Unpack ES, Extract TAs

● Sync Upgraded TAs to DS:

● Do them one at a time

● Watch for default changes and lookup overwrites

● Push TAs Out to Search/Parsing Tiers

● Use UI and Do ES Upgrade

● Remove Unnecessary TAs and Ensure Yours are Pushed

Tricks: Create a Custom TA

● It Looks Harder Than it is, but Don't Rush

● Have Your DataSource Manual Ready

● Prepare a regex parser too, as needed

● Work in Dev Environment Whenever Possible

● Copy a Similar TA

● Input the Data (Apply Necessary Parse-time Confs)

● Ensure Necessary Fields Are Present

● Ensure Necessary Tags/Eventtypes are There

● Validate Your TA (See Next Slide)

Tricks: Validate TAs

Use search or macros to verify TAs

Splunk Enterprise Security

Analytics (Unchained)

Drilldown Gets You Started (Demo)

ES Macros (Demo)

● `authentication`

● `ids_attack`

● `communicate`

● `malware`

● `proxy`

● `vulnerability`

ES Lookups (Demo)

● Assets

● | `assets` ; | inputlookup simple_asset_lookup

● | `categories` ; | inputlookup category_lookup

● Identities

● | `identities` ; | inputlookup simple_identity_lookup

● Trackers (on my)

● | `access_tracker`

● | `port_protocol_tracker`

● | `ids_attack_tracker`

Custom Analysis (Demo)

● `proxy` | search `get_subject(src, "10.11.36.23")`

● `proxy` | search NOT action="tcp_denied" [ search `proxy` | search action="tcp_denied" | dedup src | table src] | top dest by src

● `ids_attack` | search (severity="critical" OR severity=”high”) signature="dos*" `get_subject(src, "125.17.14.100")` category="dos"

Additional Resources

● docs.splunk.com - General Manuals

● docs.splunk.com/Documentation/ES - ES

● splunk-base.splunk.com - User forums

● Cheatsheet - duh!

Thank You!

ES: Trips, Tricks, Analytics (This Talk)

aplura.com/splunkconf2013

● Also:

● Best Practice PDF: aplura.com/splunkbp

● Talk: Security Analysis: aplura.com/splunklive2013

● Talk: Best Practice: aplura.com/splunklive2012

● Talk: SIEM Fails: aplura.com/lookbeforeyousim