setting up high speed logging (hsl) & configuring f5 to...
TRANSCRIPT
![Page 1: Setting up High Speed Logging (HSL) & Configuring F5 to ...carahsoft.biz/pdf/BIGIPLogging2014.pdf · Setting up High Speed Logging (HSL) & Configuring F5 to work with Splunk David](https://reader030.vdocuments.net/reader030/viewer/2022012303/5b19c8ec7f8b9a3c258cdfaf/html5/thumbnails/1.jpg)
Setting up High Speed Logging (HSL) & Configuring F5 to work with Splunk
David Perodin - FSE
![Page 2: Setting up High Speed Logging (HSL) & Configuring F5 to ...carahsoft.biz/pdf/BIGIPLogging2014.pdf · Setting up High Speed Logging (HSL) & Configuring F5 to work with Splunk David](https://reader030.vdocuments.net/reader030/viewer/2022012303/5b19c8ec7f8b9a3c258cdfaf/html5/thumbnails/2.jpg)
Agenda
Explain the necessary components for F5’s new Logging framework
Pools, Destinations, Publishers, & etc.
Demonstrate F5 and Splunk integration
Questions
![Page 3: Setting up High Speed Logging (HSL) & Configuring F5 to ...carahsoft.biz/pdf/BIGIPLogging2014.pdf · Setting up High Speed Logging (HSL) & Configuring F5 to work with Splunk David](https://reader030.vdocuments.net/reader030/viewer/2022012303/5b19c8ec7f8b9a3c258cdfaf/html5/thumbnails/3.jpg)
BIG-IP Logging
Prior to 11.3,
Logging done by different systems via different mechanisms
Configuration was totally independent of each other.
V10.1 introduced HSL support by iRules
V11.0 the HTTP Request Logging profile was introduced
V11.3
Logging systems are inter-connected
The linux host processes can now log to remote servers
![Page 4: Setting up High Speed Logging (HSL) & Configuring F5 to ...carahsoft.biz/pdf/BIGIPLogging2014.pdf · Setting up High Speed Logging (HSL) & Configuring F5 to work with Splunk David](https://reader030.vdocuments.net/reader030/viewer/2022012303/5b19c8ec7f8b9a3c258cdfaf/html5/thumbnails/4.jpg)
Logging Overview
System
AFM
High Speed DNS
Publisher Formatted
Destination HSL Dest.
Pool
![Page 5: Setting up High Speed Logging (HSL) & Configuring F5 to ...carahsoft.biz/pdf/BIGIPLogging2014.pdf · Setting up High Speed Logging (HSL) & Configuring F5 to work with Splunk David](https://reader030.vdocuments.net/reader030/viewer/2022012303/5b19c8ec7f8b9a3c258cdfaf/html5/thumbnails/5.jpg)
How is this Better?
Remote Logging available since 11.1
Available before 11.1 via the retired bigpipe CLI
No customization
• Every message sent to every entry in list of remote loggers
11.3 Filters allow separate treatment of individual daemons
11.3 Publishers allow separate treatment of different loggers
Not everyone in an organization is interested in the same logs
• System Logs to Operations
• Firewall Logs to Security Team
• Audit Logs to ???
![Page 6: Setting up High Speed Logging (HSL) & Configuring F5 to ...carahsoft.biz/pdf/BIGIPLogging2014.pdf · Setting up High Speed Logging (HSL) & Configuring F5 to work with Splunk David](https://reader030.vdocuments.net/reader030/viewer/2022012303/5b19c8ec7f8b9a3c258cdfaf/html5/thumbnails/6.jpg)
© F5 Networks, Inc 6
More versatile logging
System
AFM
High Speed DNS
Publisher Formatted
Destination (Splunk)
HSL Dest.
Pool (Splunk)
Formatted Destination (ArcSight)
Formatted Destination
(Syslog)
HSL Dest.
HSL Dest.
Pool (Syslog)
Pool (ArcSight)
![Page 7: Setting up High Speed Logging (HSL) & Configuring F5 to ...carahsoft.biz/pdf/BIGIPLogging2014.pdf · Setting up High Speed Logging (HSL) & Configuring F5 to work with Splunk David](https://reader030.vdocuments.net/reader030/viewer/2022012303/5b19c8ec7f8b9a3c258cdfaf/html5/thumbnails/7.jpg)
What's Left to Do?
Alerting
SNMP Traps
![Page 8: Setting up High Speed Logging (HSL) & Configuring F5 to ...carahsoft.biz/pdf/BIGIPLogging2014.pdf · Setting up High Speed Logging (HSL) & Configuring F5 to work with Splunk David](https://reader030.vdocuments.net/reader030/viewer/2022012303/5b19c8ec7f8b9a3c258cdfaf/html5/thumbnails/8.jpg)
Overview of Common Elements
Pool
A collection of log servers defined by IP address and port
Destination
A Destination is a Pool of log servers
May provide formatting
Publisher
A Publisher is a collection of Destinations
![Page 9: Setting up High Speed Logging (HSL) & Configuring F5 to ...carahsoft.biz/pdf/BIGIPLogging2014.pdf · Setting up High Speed Logging (HSL) & Configuring F5 to work with Splunk David](https://reader030.vdocuments.net/reader030/viewer/2022012303/5b19c8ec7f8b9a3c258cdfaf/html5/thumbnails/9.jpg)
Remote Logging Steps: Pool Creation
1. Create a Pool
2. Create a Destination
3. Create a Formatted Destination
4. Create a Publisher
5. Create tmm_filters
![Page 10: Setting up High Speed Logging (HSL) & Configuring F5 to ...carahsoft.biz/pdf/BIGIPLogging2014.pdf · Setting up High Speed Logging (HSL) & Configuring F5 to work with Splunk David](https://reader030.vdocuments.net/reader030/viewer/2022012303/5b19c8ec7f8b9a3c258cdfaf/html5/thumbnails/10.jpg)
© F5 Networks, Inc 10
Pool Creation - GUI
![Page 11: Setting up High Speed Logging (HSL) & Configuring F5 to ...carahsoft.biz/pdf/BIGIPLogging2014.pdf · Setting up High Speed Logging (HSL) & Configuring F5 to work with Splunk David](https://reader030.vdocuments.net/reader030/viewer/2022012303/5b19c8ec7f8b9a3c258cdfaf/html5/thumbnails/11.jpg)
Remote Logging Steps: Destination
1. Create a Pool
2. Create a Destination
• Create a High Speed Log (HSL) Destination
3. Create a Formatted Destination
4. Create a Publisher
5. Logging Application Steps (varies by Application)
![Page 12: Setting up High Speed Logging (HSL) & Configuring F5 to ...carahsoft.biz/pdf/BIGIPLogging2014.pdf · Setting up High Speed Logging (HSL) & Configuring F5 to work with Splunk David](https://reader030.vdocuments.net/reader030/viewer/2022012303/5b19c8ec7f8b9a3c258cdfaf/html5/thumbnails/12.jpg)
Destination
Destination Creation
A Destination is a Pool of log servers along with a Type
Configuration Elements
• Enter a unique Name
• Select a Type (see next slides)
• Remote High-Speed Log, ArcSight, Splunk or Remote Syslog
Destination Type
Unformatted
• Remote High-Speed Log (aka HSL Destination)
• Select a pool
• Formatted
• Splunk
• Requires an HSL Destination to forward too.
• ArcSight
• Requires an HSL Destination to forward too.
• Syslog
• Select a Syslog format
• And an HSL Destination
![Page 13: Setting up High Speed Logging (HSL) & Configuring F5 to ...carahsoft.biz/pdf/BIGIPLogging2014.pdf · Setting up High Speed Logging (HSL) & Configuring F5 to work with Splunk David](https://reader030.vdocuments.net/reader030/viewer/2022012303/5b19c8ec7f8b9a3c258cdfaf/html5/thumbnails/13.jpg)
Destination Creation
Go to System > Logs > Configuration > Log Destinations
![Page 14: Setting up High Speed Logging (HSL) & Configuring F5 to ...carahsoft.biz/pdf/BIGIPLogging2014.pdf · Setting up High Speed Logging (HSL) & Configuring F5 to work with Splunk David](https://reader030.vdocuments.net/reader030/viewer/2022012303/5b19c8ec7f8b9a3c258cdfaf/html5/thumbnails/14.jpg)
High-Speed Log Destination Creation
Unformatted
Must be create before formatted destinations
•
![Page 15: Setting up High Speed Logging (HSL) & Configuring F5 to ...carahsoft.biz/pdf/BIGIPLogging2014.pdf · Setting up High Speed Logging (HSL) & Configuring F5 to work with Splunk David](https://reader030.vdocuments.net/reader030/viewer/2022012303/5b19c8ec7f8b9a3c258cdfaf/html5/thumbnails/15.jpg)
Formatted Destinations
1. Create a Pool
2. Create a Destination
3. Create a Formatted Destination
• Tied to an HSL Destination
4. Create a Publisher
5. Logging Application Steps (varies by Application)
![Page 16: Setting up High Speed Logging (HSL) & Configuring F5 to ...carahsoft.biz/pdf/BIGIPLogging2014.pdf · Setting up High Speed Logging (HSL) & Configuring F5 to work with Splunk David](https://reader030.vdocuments.net/reader030/viewer/2022012303/5b19c8ec7f8b9a3c258cdfaf/html5/thumbnails/16.jpg)
Remote Syslog Destination Creation
Name your log destination
Select a syslog format
Select a High-Speed Log Destination
• Unformatted Destination you created earlier
![Page 17: Setting up High Speed Logging (HSL) & Configuring F5 to ...carahsoft.biz/pdf/BIGIPLogging2014.pdf · Setting up High Speed Logging (HSL) & Configuring F5 to work with Splunk David](https://reader030.vdocuments.net/reader030/viewer/2022012303/5b19c8ec7f8b9a3c258cdfaf/html5/thumbnails/17.jpg)
Splunk Destination Creation
Similar to create a Remote Syslog destination
Select the Splunk format
Select a High-Speed Log Destination
• Unformatted Destination you created earlier
![Page 18: Setting up High Speed Logging (HSL) & Configuring F5 to ...carahsoft.biz/pdf/BIGIPLogging2014.pdf · Setting up High Speed Logging (HSL) & Configuring F5 to work with Splunk David](https://reader030.vdocuments.net/reader030/viewer/2022012303/5b19c8ec7f8b9a3c258cdfaf/html5/thumbnails/18.jpg)
Remote Logging Steps: Publisher
1. Create a Pool
2. Create a Destination
3. Create a Formatted Destination
4. Create a Publisher
• Using one or more Destinations
5. Create tmm_filters
![Page 19: Setting up High Speed Logging (HSL) & Configuring F5 to ...carahsoft.biz/pdf/BIGIPLogging2014.pdf · Setting up High Speed Logging (HSL) & Configuring F5 to work with Splunk David](https://reader030.vdocuments.net/reader030/viewer/2022012303/5b19c8ec7f8b9a3c258cdfaf/html5/thumbnails/19.jpg)
Log Publisher
A Publisher is a collection of Destinations
Configuration Elements:
Choose a unique name for this Publisher
(Optionally) Enter a Description
Select a Destination from the available choices
![Page 20: Setting up High Speed Logging (HSL) & Configuring F5 to ...carahsoft.biz/pdf/BIGIPLogging2014.pdf · Setting up High Speed Logging (HSL) & Configuring F5 to work with Splunk David](https://reader030.vdocuments.net/reader030/viewer/2022012303/5b19c8ec7f8b9a3c258cdfaf/html5/thumbnails/20.jpg)
Support Details - Uneven Load Balancing
Load balancing across Pools of remote logging servers
BIG-IP follows the connection/session
BIG-IP doe not load balance by message
At low volumes of logging uneven log message counts will be seen.
• For example in testing or performing a POC.
HSL will not make a load balancing decision
Until it runs out of bandwidth to the selected pool member.
Or there is a change in server response
![Page 21: Setting up High Speed Logging (HSL) & Configuring F5 to ...carahsoft.biz/pdf/BIGIPLogging2014.pdf · Setting up High Speed Logging (HSL) & Configuring F5 to work with Splunk David](https://reader030.vdocuments.net/reader030/viewer/2022012303/5b19c8ec7f8b9a3c258cdfaf/html5/thumbnails/21.jpg)
Publisher local-db-publisher
Used by the legacy logging system
Local logging places an I/O load on the BIG-IP
Should not be used, can have a significant impact
![Page 22: Setting up High Speed Logging (HSL) & Configuring F5 to ...carahsoft.biz/pdf/BIGIPLogging2014.pdf · Setting up High Speed Logging (HSL) & Configuring F5 to work with Splunk David](https://reader030.vdocuments.net/reader030/viewer/2022012303/5b19c8ec7f8b9a3c258cdfaf/html5/thumbnails/22.jpg)
Previous Remote Logging Option
This screen introduced in V11.1
Does not load balance
All Syslog servers in the list receive a copy of the message
![Page 23: Setting up High Speed Logging (HSL) & Configuring F5 to ...carahsoft.biz/pdf/BIGIPLogging2014.pdf · Setting up High Speed Logging (HSL) & Configuring F5 to work with Splunk David](https://reader030.vdocuments.net/reader030/viewer/2022012303/5b19c8ec7f8b9a3c258cdfaf/html5/thumbnails/23.jpg)
11.3 System Logging - A New Paradigm
Required: elements described previously
Pool
Destination
Publisher
What is unique is the tmm_filter
![Page 24: Setting up High Speed Logging (HSL) & Configuring F5 to ...carahsoft.biz/pdf/BIGIPLogging2014.pdf · Setting up High Speed Logging (HSL) & Configuring F5 to work with Splunk David](https://reader030.vdocuments.net/reader030/viewer/2022012303/5b19c8ec7f8b9a3c258cdfaf/html5/thumbnails/24.jpg)
tmm_filter
Under System > Logs > Configuration > Log Filers
Can create custom filters
Name
Description (optional)
Severity
• Default is Debug
Source
• List of processes
• Defaults to all
Message ID
Log Publisher
![Page 25: Setting up High Speed Logging (HSL) & Configuring F5 to ...carahsoft.biz/pdf/BIGIPLogging2014.pdf · Setting up High Speed Logging (HSL) & Configuring F5 to work with Splunk David](https://reader030.vdocuments.net/reader030/viewer/2022012303/5b19c8ec7f8b9a3c258cdfaf/html5/thumbnails/25.jpg)
Severity
Filter base on severity
Name (required)
Description (optional)
Severity
![Page 26: Setting up High Speed Logging (HSL) & Configuring F5 to ...carahsoft.biz/pdf/BIGIPLogging2014.pdf · Setting up High Speed Logging (HSL) & Configuring F5 to work with Splunk David](https://reader030.vdocuments.net/reader030/viewer/2022012303/5b19c8ec7f8b9a3c258cdfaf/html5/thumbnails/26.jpg)
Source
Filter base on process
Source
• Select from the list of processes
• Defaults to all
![Page 27: Setting up High Speed Logging (HSL) & Configuring F5 to ...carahsoft.biz/pdf/BIGIPLogging2014.pdf · Setting up High Speed Logging (HSL) & Configuring F5 to work with Splunk David](https://reader030.vdocuments.net/reader030/viewer/2022012303/5b19c8ec7f8b9a3c258cdfaf/html5/thumbnails/27.jpg)
11.3 System Logging
Filter base on Message ID
Message ID
Log Publisher
• Message destination(s)
![Page 28: Setting up High Speed Logging (HSL) & Configuring F5 to ...carahsoft.biz/pdf/BIGIPLogging2014.pdf · Setting up High Speed Logging (HSL) & Configuring F5 to work with Splunk David](https://reader030.vdocuments.net/reader030/viewer/2022012303/5b19c8ec7f8b9a3c258cdfaf/html5/thumbnails/28.jpg)
© F5 Networks, Inc 28
Interaction of Legacy Paradigm and Filters
Log Messages
Filter match
No Filter
Publisher
Syslog (legacy)
![Page 29: Setting up High Speed Logging (HSL) & Configuring F5 to ...carahsoft.biz/pdf/BIGIPLogging2014.pdf · Setting up High Speed Logging (HSL) & Configuring F5 to work with Splunk David](https://reader030.vdocuments.net/reader030/viewer/2022012303/5b19c8ec7f8b9a3c258cdfaf/html5/thumbnails/29.jpg)
© F5 Networks, Inc 29
All Logging Done Off the BIG-IP
Log Messages
Filter match
Publisher
Publisher (none)
Filter match
all debug
Syslog (legacy)
Nothing
unmatched
![Page 30: Setting up High Speed Logging (HSL) & Configuring F5 to ...carahsoft.biz/pdf/BIGIPLogging2014.pdf · Setting up High Speed Logging (HSL) & Configuring F5 to work with Splunk David](https://reader030.vdocuments.net/reader030/viewer/2022012303/5b19c8ec7f8b9a3c258cdfaf/html5/thumbnails/30.jpg)
DANGEROUS DEFAULTS
Beware the default severity 'debug' and default source 'all'
![Page 31: Setting up High Speed Logging (HSL) & Configuring F5 to ...carahsoft.biz/pdf/BIGIPLogging2014.pdf · Setting up High Speed Logging (HSL) & Configuring F5 to work with Splunk David](https://reader030.vdocuments.net/reader030/viewer/2022012303/5b19c8ec7f8b9a3c258cdfaf/html5/thumbnails/31.jpg)
Thank You! Thank You!
![Page 32: Setting up High Speed Logging (HSL) & Configuring F5 to ...carahsoft.biz/pdf/BIGIPLogging2014.pdf · Setting up High Speed Logging (HSL) & Configuring F5 to work with Splunk David](https://reader030.vdocuments.net/reader030/viewer/2022012303/5b19c8ec7f8b9a3c258cdfaf/html5/thumbnails/32.jpg)