splunk dashboarding & universal vs. heavy forwarders
TRANSCRIPT
![Page 1: Splunk Dashboarding & Universal Vs. Heavy Forwarders](https://reader036.vdocuments.net/reader036/viewer/2022062900/58e9ff111a28ab88748b5355/html5/thumbnails/1.jpg)
Copyright © 2017 Splunk Inc.
Splunk User Group EdinburghAwesome Dashboarding & UF Vs. HFFebruary 2017
![Page 2: Splunk Dashboarding & Universal Vs. Heavy Forwarders](https://reader036.vdocuments.net/reader036/viewer/2022062900/58e9ff111a28ab88748b5355/html5/thumbnails/2.jpg)
2
Introduction - Harry McLaren● Alumnus of Edinburgh Napier● Senior Security Consultant at ECS – Role: Specialist Splunk Consultant & Enablement Lead– Specialism: Enterprise Security (SIEM) / Complex Deployments
● Splunk User Group Edinburgh: Leader / Founder
![Page 3: Splunk Dashboarding & Universal Vs. Heavy Forwarders](https://reader036.vdocuments.net/reader036/viewer/2022062900/58e9ff111a28ab88748b5355/html5/thumbnails/3.jpg)
3
Introduction to ECSStrategic Splunk Partner - UK – Type: Security / IT Operations / Managed Services– Awards: Splunk Revolution Award & Splunk Partner of the Year 2016
![Page 4: Splunk Dashboarding & Universal Vs. Heavy Forwarders](https://reader036.vdocuments.net/reader036/viewer/2022062900/58e9ff111a28ab88748b5355/html5/thumbnails/4.jpg)
4
![Page 5: Splunk Dashboarding & Universal Vs. Heavy Forwarders](https://reader036.vdocuments.net/reader036/viewer/2022062900/58e9ff111a28ab88748b5355/html5/thumbnails/5.jpg)
5
Agenda
• Housekeeping: Overview & House Rules
• Presentation & Demo: Creating Awesome Dashboards
• Group Discussion: Sharing Dashboarding Tips & Tricks
• Presentation: Universal vs. Heavy vs. Intermediate Forwarders
• Group Discussion: Latest Splunk Challenges / Solutions
![Page 6: Splunk Dashboarding & Universal Vs. Heavy Forwarders](https://reader036.vdocuments.net/reader036/viewer/2022062900/58e9ff111a28ab88748b5355/html5/thumbnails/6.jpg)
6
Splunk [Official] User Group“The overall goal is to create an authentic, ongoing
user group experience for our users, where they contribute and get involved”
● User Lead Technical Discussions
● Sharing Environment
● Build Trust
● No Sales!
![Page 7: Splunk Dashboarding & Universal Vs. Heavy Forwarders](https://reader036.vdocuments.net/reader036/viewer/2022062900/58e9ff111a28ab88748b5355/html5/thumbnails/7.jpg)
Creating Awesome DashboardsRobert Williamson
![Page 8: Splunk Dashboarding & Universal Vs. Heavy Forwarders](https://reader036.vdocuments.net/reader036/viewer/2022062900/58e9ff111a28ab88748b5355/html5/thumbnails/8.jpg)
Robert Williamson
Alumnus of Edinburgh Napier university
IBM - Security Specialist
ECS - SOC Analyst, Senior SOC Analyst and Security Consultant
8
![Page 9: Splunk Dashboarding & Universal Vs. Heavy Forwarders](https://reader036.vdocuments.net/reader036/viewer/2022062900/58e9ff111a28ab88748b5355/html5/thumbnails/9.jpg)
What is a Dashboard?
9
![Page 10: Splunk Dashboarding & Universal Vs. Heavy Forwarders](https://reader036.vdocuments.net/reader036/viewer/2022062900/58e9ff111a28ab88748b5355/html5/thumbnails/10.jpg)
Creating a Dashboard
10
![Page 11: Splunk Dashboarding & Universal Vs. Heavy Forwarders](https://reader036.vdocuments.net/reader036/viewer/2022062900/58e9ff111a28ab88748b5355/html5/thumbnails/11.jpg)
Visualizations
11
![Page 12: Splunk Dashboarding & Universal Vs. Heavy Forwarders](https://reader036.vdocuments.net/reader036/viewer/2022062900/58e9ff111a28ab88748b5355/html5/thumbnails/12.jpg)
Table Formats
12
![Page 13: Splunk Dashboarding & Universal Vs. Heavy Forwarders](https://reader036.vdocuments.net/reader036/viewer/2022062900/58e9ff111a28ab88748b5355/html5/thumbnails/13.jpg)
Single Value – Colours
13
![Page 14: Splunk Dashboarding & Universal Vs. Heavy Forwarders](https://reader036.vdocuments.net/reader036/viewer/2022062900/58e9ff111a28ab88748b5355/html5/thumbnails/14.jpg)
Form Elements Within Panels
14
![Page 15: Splunk Dashboarding & Universal Vs. Heavy Forwarders](https://reader036.vdocuments.net/reader036/viewer/2022062900/58e9ff111a28ab88748b5355/html5/thumbnails/15.jpg)
Choropleth Maps
15
![Page 16: Splunk Dashboarding & Universal Vs. Heavy Forwarders](https://reader036.vdocuments.net/reader036/viewer/2022062900/58e9ff111a28ab88748b5355/html5/thumbnails/16.jpg)
I Could go on... But how is it done?
16
![Page 17: Splunk Dashboarding & Universal Vs. Heavy Forwarders](https://reader036.vdocuments.net/reader036/viewer/2022062900/58e9ff111a28ab88748b5355/html5/thumbnails/17.jpg)
Simple XML
17
![Page 18: Splunk Dashboarding & Universal Vs. Heavy Forwarders](https://reader036.vdocuments.net/reader036/viewer/2022062900/58e9ff111a28ab88748b5355/html5/thumbnails/18.jpg)
18
![Page 19: Splunk Dashboarding & Universal Vs. Heavy Forwarders](https://reader036.vdocuments.net/reader036/viewer/2022062900/58e9ff111a28ab88748b5355/html5/thumbnails/19.jpg)
Dashboard Competition
![Page 20: Splunk Dashboarding & Universal Vs. Heavy Forwarders](https://reader036.vdocuments.net/reader036/viewer/2022062900/58e9ff111a28ab88748b5355/html5/thumbnails/20.jpg)
Grab you phone and go to:http://splunk.com/shake
20
![Page 21: Splunk Dashboarding & Universal Vs. Heavy Forwarders](https://reader036.vdocuments.net/reader036/viewer/2022062900/58e9ff111a28ab88748b5355/html5/thumbnails/21.jpg)
Sharing Dashboarding Tips & TricksGroup Discussion
![Page 22: Splunk Dashboarding & Universal Vs. Heavy Forwarders](https://reader036.vdocuments.net/reader036/viewer/2022062900/58e9ff111a28ab88748b5355/html5/thumbnails/22.jpg)
Universal Vs. Heavy ForwardersHarry McLarenBased on Darren Dance’s Blog
![Page 23: Splunk Dashboarding & Universal Vs. Heavy Forwarders](https://reader036.vdocuments.net/reader036/viewer/2022062900/58e9ff111a28ab88748b5355/html5/thumbnails/23.jpg)
23
Universal Vs. Heavy (+ Intermediate) Forwards
Universal Forwarder
HeavyForwarder
IntermediateForwarder
● Smallest Footprint● Standard Data Collection● Un-Parsed /
No Event Breaking
● Larger Footprint● Full Splunk Enterprise
Binary Install● Allows Filtering at
Source / In-Flight
● UF or HF Binaries● Aggregation Layer● Artificial Bottleneck● Performance Impact
![Page 24: Splunk Dashboarding & Universal Vs. Heavy Forwarders](https://reader036.vdocuments.net/reader036/viewer/2022062900/58e9ff111a28ab88748b5355/html5/thumbnails/24.jpg)
24
Heavy Forwarders Are[n’t] Awesome!The use of Heavy Forwarders were once commonly advised, but times change…
● Previous advice for using Heavy Forwarders – Filtering of data is best done at source and HF are required as UF cannot parse. – Use for aggregation layer for central management of data flows. ‣ Can cause data imbalance on the indexing tier that will reduce search performance.
● Reasons for NOT using Heavy Forwarder– Filter data at the Indexers. Greater use of compute resources / more performant.– Reduces network usage / IO by a significant degree. – Reduces the time from event generation to search availability. – Segmentation doesn’t always reduce threat vector for application exploitation.
![Page 25: Splunk Dashboarding & Universal Vs. Heavy Forwarders](https://reader036.vdocuments.net/reader036/viewer/2022062900/58e9ff111a28ab88748b5355/html5/thumbnails/25.jpg)
25
Artificial Bottleneck with IF
![Page 26: Splunk Dashboarding & Universal Vs. Heavy Forwarders](https://reader036.vdocuments.net/reader036/viewer/2022062900/58e9ff111a28ab88748b5355/html5/thumbnails/26.jpg)
26
Performance Impact Test Setup: File Contained 367,463,625 Events
Indexer Acknowledgement
Network Data Transferred (GB)
Network Speed Average (KBps)
Indexing Speed Average (KBps)
Duration (Secs)
HeavyYes 39.1 1,941 5,092 21,151
No 38.4 1,922 5,139 20,998
UniversalYes 6.5 863 14,344 7,923
No 6.4 1,015 17,466 6,662
![Page 27: Splunk Dashboarding & Universal Vs. Heavy Forwarders](https://reader036.vdocuments.net/reader036/viewer/2022062900/58e9ff111a28ab88748b5355/html5/thumbnails/27.jpg)
27
Performance Impact
Key Takeaways● The amount of data sent over the network was approximately 6 times
lower with the Universal Forwarder.● The amount of data indexed per second was approximately 3 times
higher when collected by a Universal Forwarder.● The total data set was indexed approximately 6 times quicker when
collected by the Universal Forwarder.
![Page 28: Splunk Dashboarding & Universal Vs. Heavy Forwarders](https://reader036.vdocuments.net/reader036/viewer/2022062900/58e9ff111a28ab88748b5355/html5/thumbnails/28.jpg)
28
Ideal Distribution with UF
![Page 29: Splunk Dashboarding & Universal Vs. Heavy Forwarders](https://reader036.vdocuments.net/reader036/viewer/2022062900/58e9ff111a28ab88748b5355/html5/thumbnails/29.jpg)
29
What About Network Segmentation?
● Limited Reduction to Application Threat Vector (UF > IF > IX)– If the Splunk software on the IF are vulnerable, then the same exploit could be
used to pivot into the next network layer anyway. ● Network Load
– If using a HF to aggregate the forwarder traffic, the additional network load could be upwards of 6x more than if UF directly to Indexers (Raw Vs. Parsed Data)
![Page 30: Splunk Dashboarding & Universal Vs. Heavy Forwarders](https://reader036.vdocuments.net/reader036/viewer/2022062900/58e9ff111a28ab88748b5355/html5/thumbnails/30.jpg)
30
Exceptions to UF > HFSome exceptions to using Universal Forwarders over Heavy Forwarders
● Special App Requirements – DB Connect / eStreamer / Opsec LEA / Etc.
● Modify In-Flight Events (Parsed Data Stream)– Change data before it leaves a specific environment (pattern replacement).
● Routing Based on Event Contents– Route data based on criteria such as source or type of event.
![Page 31: Splunk Dashboarding & Universal Vs. Heavy Forwarders](https://reader036.vdocuments.net/reader036/viewer/2022062900/58e9ff111a28ab88748b5355/html5/thumbnails/31.jpg)
31
Cloud Architecture
![Page 32: Splunk Dashboarding & Universal Vs. Heavy Forwarders](https://reader036.vdocuments.net/reader036/viewer/2022062900/58e9ff111a28ab88748b5355/html5/thumbnails/32.jpg)
32
Any Questions?
![Page 33: Splunk Dashboarding & Universal Vs. Heavy Forwarders](https://reader036.vdocuments.net/reader036/viewer/2022062900/58e9ff111a28ab88748b5355/html5/thumbnails/33.jpg)
33
Updates Announced at .conf 2016● Introducing Splunk Enterprise 6.5 - Available Now
‣ Splunk ML Toolkit: Guided workbench and SPL extensions to help you create and operationalize your own custom analytics based on your choice of algorithms.
‣ Tables: New feature that lets you create and analyse tabular data views without using SPL.
‣ Hadoop Data Roll: Gives you another way to reduce historical data storage costs while keeping full search capability.
● Premium Apps - New Releases:– Splunk Enterprise Security [Minor Release] – Splunk IT Service Intelligence [Major Release]– Splunk User Behaviour Analytics [Major Release]
![Page 34: Splunk Dashboarding & Universal Vs. Heavy Forwarders](https://reader036.vdocuments.net/reader036/viewer/2022062900/58e9ff111a28ab88748b5355/html5/thumbnails/34.jpg)
34
Get Involved!● Splunk User Group Edinburgh– https://usergroups.splunk.com/group/splunk-user-group-edinburgh.html– https://www.linkedin.com/groups/12013212
● Splunk’s Slack Group– Register via www.splunk402.com/chat – Channel: #edinburgh
● Present & Share at the User Group?Connect:‣ Harry McLaren | [email protected] | @cyberharibu | harrymclaren.co.uk‣ ECS | [email protected] | @ECS_IT | ecs.co.uk
![Page 35: Splunk Dashboarding & Universal Vs. Heavy Forwarders](https://reader036.vdocuments.net/reader036/viewer/2022062900/58e9ff111a28ab88748b5355/html5/thumbnails/35.jpg)
Thank You