dissecting firepower-ftd · virtual fdm • allows users to ... config profile_preprocs: print all,...

Dissecting Firepower-FTD & Firepower-Services “Design & Troubleshooting”

Foster Lipkey, Technical Leader

Veronika Klauzova, TAC Tech Lead

BRKSEC-3455

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public

Cisco Spark

Questions? Use Cisco Spark to communicate with the speaker after the session

1. Find this session in the Cisco Live Mobile App

2. Click “Join the Discussion”

3. Install Spark or go directly to the space

4. Enter messages/questions in the space

How

cs.co/ciscolivebot#BRKSEC-3455

• Introduction

• Updated FTD Packet Flow

• Data-Path Improvements

• Firepower New Features in 6.2.2.X

• Best Practices for Deployments

• Troubleshooting Tools

• Exciting Real-World Use-Cases

• Conclusions

Agenda


Your presenter throughout FTD journey

• Firepower TAC TL

Foster Lipkey

• Snort Expert

• Sourcefire Veteran

• Automation Enthusiast

BRKSEC-3455 5


Your presenter throughout FTD journey

• Firepower TAC engineer

Veronika Klauzova

• Love to explore

Cisco technologies

• Passionate Linux Admin

BRKSEC-3455 6

Hardware & Software Review


NGFW evolution

BRKSEC-3455 8


What platforms can run FTD Software

ASA 5500X-Series (5506X-5555X with SSD)

BRKSEC-3455 9



Firepower 2100 series

BRKSEC-3455 10



Front view

Rear view

PowerConsole

MGMT

8 x optic SFP+ ports

2 x 2.5” SSD Bays

2 x Power Supply Module Bays6 x Hot-Swap Fans units

2x optional NetMods

Firepower 4100 series

BRKSEC-3455 11

Updated FTD Packet Flow


Firepower Threat Defense – high level

DETECTION ENGINE / Snort

DATA-PATH / LINA

Packet Data Transport System (PDTS)

FXOS

BRKSEC-3455 16


Firepower 2100 architecture overview

BRKSEC-3455 17


Firepower 9300/4100 architecture overview

BRKSEC-3455 18


FTD Packet-Flow

Data-Path / LINA

RX

L3, L2

hopsPre-Filter L3/L4

ACL

Egress

Interface NAT

TX

ALGchecks

Ingress

Interface

Existing

Conn

Detection Engine / Snort

PDTSYES

NO

DAQ

VPN DecryptQoS, VPN Encrypt

Lina rule-id matched

BRKSEC-3455 19


Detection Engine/ Snort - Architecture

LIN

A /

Data

-Path

BRKSEC-3455 20

Data-Path Improvements


Data-Path improvements / Safe Guards

Snort Fail Open When Busy

• If the buffer going into Snort is 85% full, new flows will be bypassed

Snort Fail Open When Down

• When Snort goes does due to restart for policy deploy, or for any other reason new flows will be bypassed

BRKSEC-3455 22


Data-Path improvements / Safe Guards

Automation Application Bypass

• If traffic enters Snort through the buffer and does not provide a verdict back to LINA within configured threshold, Snort is restarted and a core file is generated

Device > Device Management [Edit] > Device tab

BRKSEC-3455 23

Show Time


Snort reload instead of restart

• As of 6.2.2 following changes would not cause Snort to be restarted

• This applies to all FTD devices managed by FMC 6.2.2

Policy changes Policy action

URL Refer to URL categories for the first time in AC rules or remove all

existing references to URL categories

Application ID Turn on/off Application ID

Intrusion Policy Add or Delete Intrusion Polices in AC rules, or Edit Intrusion Policy

NAP policy Attach a NAP policy for the first time to AC Policy

Simple SRU update Typical rule updates without Shared Object (SO) / binary rule updates

Security Intelligence Changes to Whitelist/Blacklist of URL, DNS entries

BRKSEC-3455 25


Other snort major updates

• Changes to application detectors display warnings

• Break HA operation restart snort/s (warning displayed)

• Memory allocation changed

• SRU simple rule changes does not cause snort restart, but binary objects do

• Binary changes are not that frequent

• Whether snort would affect it depends on system resources

BRKSEC-3455 26


• When Snort goes down connections with Allow verdict are preserved in LINA

• Snort does NOT do a mid-session pickup on preserved flows on coming up

• Does NOT protect against new flows while Snort is down

• 6.2.0.2 Feature Introduction

• Can be enabled/disabled from CLISH:configure snort preserve-connection enable/disable

Snort Preserve-Connection

BRKSEC-3455 27


Minimalize network disruption during policy deployment

• Snort restart behavior depends on Advance settings in Access Control Policy

• TAC highly recommend to enable:

• Inspect traffic during policy apply = Yes

• Without this option Snort always restarts during policy deployment

BRKSEC-3455 28


Snort Restart & Reload Architecture

BRKSEC-3455 29

Show Time

Firepower New Features in 6.2.2.X


New Signed Software Update/Upgrade images

• Signed images were introduced in 6.2.1

• Signed images are the .RHEL.tar files (caution: DO NOT UNTAR THEM!)

• Managed FTD device can be upgraded only after FMC is upgraded

• FTD on platforms 4100 and 9300 series needs to have upgraded FXOS software via Firepower Chassis Manager prior FTD upgrade to 6.2.2 version

• To update an FMC from 6.2.0 to 6.2.2 release an unsigned upgrade package need to be used (.sh file)

Platform Current Version Destination Version Package name to be used

FMC 6.2.0 6.2.2 Sourcefire_3D_Defense_Center_S3_Upgrade-6.2.2-81.sh

FMC 6.2.1 6.2.2 Sourcefire_3D_Defense_Center_S3_Upgrade-6.2.2-81.sh.REL.tar

BRKSEC-3455 32


Virtual FDM

• Allows users to manage virtual platforms using on-box management

• Only fresh installation to 6.2.2 enables FDMv management option

• Initial setup can be done once, it cannot be relaunched

• Adding/removing interfaces on already running FTDv requires deregistration of management (all configuration will be lost!)

BRKSEC-3455 33


Threat Intelligence Director

• Consumes third-party cyber threat intelligence

• Requirements:

• FMC and FTD running 6.2.2

• 15 GB of memory

• Protect license (IPv4, IPv6, Domain and URL detection)

• Malware license (SHA-256 detection)

• Terminology

• STIX – Structured Threat Intelligence eXpression

• TAXII – transport mechanism for STIX

• TID is activated under Access Control Policy Advanced tab

• TID correlation for incident generation is dependent on an exact match!

BRKSEC-3455 34


TID High-Level Architecture

NGFW / NGIPS

(manage device)

Observables

Sftunnnel (TCP 8305)

STIX TAXII Flat files

Third-Party Cyber Security Intelligence

Cisco TID on FMC

Syncd.pl

Can take up to 20 minutes!

BRKSEC-3455 35


TID Troubleshooting

Observables type File location

IPv4 and IPv6 addresses /ngfw/var/sf/iprep_download

Domain names /ngfw/var/sf/sidns_download

URL’s /ngfw/var/sf/siurl_download

SHA-256 hashes /ngfw/var/sf/sifile_download

BRKSEC-3455 36


API bulk rule access insertion, yay!

• Old behavior: one AC rule can be imported at the time

• New behavior: we can insert up to 1000 rules within same API request!

• How cool is that?

• We can insert rules at specific location (rule number or within specific category/section)

• After rule insertion, other rules are automatically reordered

• Rest API can handle if other user is already modifying the same rule set

• When no position of the rule is defined, it goes to the end of ACP

BRKSEC-3455 37

Best Practices for Deployments(security is our priority)


VPN deployment on FTD: things that you might have missed!

FMC

NGFW

Cisco network

The Internet

FTP Servers

insideoutside

Cisco Employee

working from home

attacker

Should been never been allowed

Anyconnect (encrypted session)

Clear-text / un-authenticated session

BRKSEC-3455 39


Is your network protected?

BRKSEC-3455 40


FTD / IPS is dropping packets … HELP!

• FTD detection engine / IPS bottleneck causing throughput issues

• Do we have enough processing power / right hardware?

• What is traffic pattern / volume? (the type, size and protocol of packet)

• Why not simply enable all of the rules?

• …. Ok, now really, how many Snort signatures are enabled?

• expensive signatures & local rule

• IPS alerting load (processing and disk operations)

• Expensive work on preprocessors

BRKSEC-3455 42


Tuning IPS rules #(TAC tip & trick)

• Use case: poor performance with default IPS policy baseline for FTP traffic

• Simplified topology:

client (Windows 10) ---1Gbps --- FTD 9300 ---1Gbps --- server (Windows 10)

Performance measurement results with default policy

~ 380 Mbps

Performance measurement after IPS rule tuning

~ 970 Mbps

BRKSEC-3455 43


Full example: performance numbers from field/lab testingMode Protocol Configuration Throughput

Transparent FTP (Filezilla 3.29.0 ) Pre-filter policy with Fast-path rule for TCP ports 20 and 21 ~979 Mbps

Access Control Policy, Allow rule for TCP ports 20 and 21,

IPS connectivity over Security

~650 Mbps


IPS Balanced Security and Connectivity

~380 Mbps


IPS Security over Connectivity

~340 Mbps


IPS Maximum detection

~320 Mbps


IPS tuned (base no rules active + 51 active rules)

Filter used: ftp metadata:"security-ips drop"

~971 Mbps


IPS tuned (base no rules active + 51 active rules)

Filter used: ftp metadata:"security-ips drop"

+ File policy with application protocol FTP (detect all file

types and block malware executable’s with local malware

analysis)

~800 Mbps

BRKSEC-3455 44


Low IPS performance? … rule it out by FTD rule profiling!

Edit /ngfw/var/sf/detection_engines/<uuid>/ advanced/perf_monitor.conf

config profile_rules: print all, sort avg_ticks, filename /ngfw/var/log/profiling-rules.log

config profile_preprocs: print all, sort avg_ticks, filename /ngfw/var/log/profiling-preprocs.log

Restart Snort

pmtool restartbytype snort

Start rule profiling

> system support run-rule-profiling

BRKSEC-3455 45


Low IPS performance? … rule it out by FTD rule profiling!

BRKSEC-3455 46


Performance graphs from the WebUI

Why does Bytes/Packet matter?

BRKSEC-3455 47


Reassembly cost

Posted throughput ratings for the Firepower appliances are usually rated at 1518 bytes packets. Smaller packets results in more processing.

1MB of traffic with 1518 bytes/packets = ~ 658 packets

1MB of traffic with 400 bytes/packet = ~ 2500 packets

Every packet header must be evaluated and the packet has to be placed into the buffer for re-assembly. The larger number of packets to process requires more CPU time.

BRKSEC-3455 48


Let’s talk about the elephant in the room…

Large flows are generally related backup, database replication, etc. which usually does not require inspection

Sort Analysis > Connections for connection size to find top talkers

Once we determine the top talkers, and confirm they can be safely ignored, we create trust rule for the IP conversations.

Mitigations IAB / Pre-Filter fast-path

BRKSEC-3455 49


Sizing your NGFW / NGIPS – throughput considerations

• Number of Snort instances per FTD platform

Platform Snort Instances Platform Snort Instances Platform Snort

instances

Firepower 2110 6 Firepower 4110 11 Firepower 9300 SM-24 24



Firepower 2140 26 Firepower 4150 48 - -

For YourReference

Enabling File-Inspection will change these values > pmtool show affinity

BRKSEC-3455 50

Troubleshooting Tools


What are main FTD processes and what they do?snort inspects network traffic (pass,

block and alert)

sftunnel secure tunnel between

managed device and FMC

ids_event_processor sends intrusion events to

managing device (FMC)

diskmanager,

Pruner

managing disk space and

clean up old files

ids_event_alerter sends intrusion events to

Syslog or SNMP serverLina Responsible for Firewall

functionality like ACL, NAT, Routing etc.

wdt-util used for fail-to-wire /

hardware bypass

Snmpd,

ntpd

SNMP monitoring,responsible for time

synchronization

SFDataCorrelator processing events pm (process

manager)

responsible for launching

and monitoring of all FTD

relevant processes and

restarting them in case of

failure

BRKSEC-3455 52


Process Management - basics

Process name

Category

Status

Process ID

FTD Root CLI:

ftd-vklauzov:/# pmtool status | grep " - " | head

SFDataCorrelator (normal) - Running 15278

mysqld (system,gui,mysql) - Running 15109

httpsd (system,gui) - Waiting

sftunnel (system) - Running 19857

BRKSEC-3455 53


Process Management - basicsFMC Root CLI:

root@fmc-2:/# pmtool disablebyid sftunnel

root@fmc-2:/# pmtool status | grep " - " | grep sftunnel

sftunnel (system) - User Disabled

root@fmc-2:/# pmtool enablebyid sftunnel

root@fmc-2:/# pmtool status | grep " - " | grep sftunnel

sftunnel (system) - Running 1720

BRKSEC-3455 54


Data-path and Snort capture points

Detection Engine / Snort

DATA-PATH

data-path inbound

data-path outbound

snort inbound/outbound

1.

2.

3.firepower# capture in

firepower# capture out

> capture-traffic

BRKSEC-3455 55


Data-path inbound/outbound - The Wires Never Lie!

firepower# capture in interface INSIDE match icmp any any trace detail

Capture nameInterface name

protocol

SourceDestination

Data-path/lina (diagnostic cli):

BRKSEC-3455 56


Data-path – stop and delete captures

firepower# no capture in

Delete packet captures

firepower# no capture in interface inside

Stop packet captures

BRKSEC-3455 57

Show Time


Snort Capture - The Wires Never Lie! (1)CLISH:

> capture-traffic

Options: -s 0 -w capture.pcap icmp and host 172.16.1.17

IP 172.16.1.17 > 20.20.20.100: ICMP echo request,id 24538,seq 1,length 64

Berkeley Packet Filter syntax – same as for tcpdump capturing tool

-s 0 means snaplength, in other words no limit for packet size

-w filename.pcap indicates to which file you want to write output of data captured by specified filter

capture is written to /ngfw/var/common/ folder

Copy file out to SCP server:

file secure-copy <IP address of server> <username> <location where to copy the file> capture.pcap

BRKSEC-3455 61


> capture-traffic

Options: -s 0 -v -n -e (icmp and host 172.16.2.11) or

(vlan and icmp and host 172.16.2.11)

00:50:56:b6:0b:33 > 58:97:bd:b9:73:ee, ethertype 802.1Q (0x8100), length 78: vlan 208,

p 0, ethertype IPv4, (tos 0x0, ttl 128, id 5366, offset 0, flags [none], proto ICMP

(1), length 60)

Snort Capture - The Wires Never Lie! (2)

CLISH:

firepower# sh cap inside

802.1Q vlan#208 P0 172.16.2.11 >

20.20.20.11: icmp: echo request

LINA CLI:

firepower# sh cap outside

172.16.2.11 > 20.20.20.11: icmp: echo

request

IN OUT

LINA CLI:

NON-VLAN TAGGED TRAFFIC VLAN TAGGED TRAFFIC

BRKSEC-3455 62


Which ACP rule is being evaluated?

>system support firewall-engine-debug

Please specify an IP protocol: icmp

Please specify a client IP address: 172.16.1.17

Please specify a server IP address: 20.20.20.100

Monitoring firewall engine debug messages172.16.1.17-8 >

20.20.20.100-0 1 AS 1 I 44 New session

172.16.1.17-8 > 20.20.20.100-0 1 AS 1 I 44 using HW or preset

rule order 2, 'allow and inspect', action Allow and prefilter

rule 0

172.16.1.17-8 > 20.20.20.100-0 1 AS 1 I 44 allow action

• Tool that provides the Access Control Rule evaluation status for each flow as we receive packets in real time.

• NGFW debug needs to have specified at least one filtering condition.

BRKSEC-3455 63

Show Time


Access Control Policy Rule Hit Counters> show access-control-config

===================[ ciscolive ]====================

Description :

Default Action : Allow

Default Policy : Balanced Security and Connectivity

Logging Configuration

DC : Disabled

Beginning : Disabled

End : Disabled

Rule Hits :

Variable Set : Default-Set

... (output omitted) ...

# watch ´/usr/local/sf/bin/sfcli.pl show firewall | grep "ciscolive\| Rule\:\|Rule Hits "´

===================[ ciscolive ]====================

Rule Hits :

------------------[ Rule: allow ]-------------------

Rule Hits : 14

Policy name

Rule name

10141926

10141926

BRKSEC-3455 65


ACP Rule Hit Counters – FMC WebUI

• Analysis -> Custom -> Custom Workflows -> Create Custom Workflow and use Table “Connection Events”

• Add page and fill in fields like: “Access Control Policy”, “Access Control Rule”, “Count”, “Initiator IP”, “Responder IP”

• Add Table view

BRKSEC-3455 66


ACP Rule Hit Counters – FMC WebUI vs CLISH

Why the hit counters do not match?

> show access-control-config

------[ Rule: DNS and icmp ]------

Action : Allow

Destination Ports : protocol 6, port 53

protocol 17, port 53

protocol 1

protocol 6, port 80

Logging Configuration

DC : Enabled

Beginning : Enabled

End : Enabled

Rule Hits : 28

Variable Set : Default-Set

(truncated)

BRKSEC-3455 67


Capture With Trace – GUI

Quickly Identify where in the data-path the traffic is impacted

BRKSEC-3455 68

Show Time


CLI Analyzer

Contextual help and highlighting

Embedded Intelligence

File Analysis

BRKSEC-3455 71

Show Time


I’m a trouble-shooter …now

Firewall-Engine-DebugCapture-traffic

LIN

A /

Data

-Path

System Support Trace

Capture w/ trace

BRKSEC-3455 73

Exciting Real-World Use-Cases


Real World Scenario

Following a migration from ASAs to FTDs on pair of boarder firewalls intermittent

outages occur.

Intermittent network outages following migration to FTD

BRKSEC-3455 75


Real World Scenario – Using our tools

Symptoms:

• Migration from ASAs to FTDs results in outage under load.

• When placing ASAs back inline outage does not occur

Troubleshooting:

• Performance review

• Capture w/ Trace

• Packet Capture with FTDs inline

• Packet Capture with ASAs inline

• Compared the packet captures

Root Cause

• Sysopt connection tcpmss set to 0

• Changed to 0 by adding jumbo frames to interface

Working:

Failed:

Sometimes its what the FW didn’t do that counts

BRKSEC-3455 76


Real World Scenario

HARDWARE ERROR ON LCD

HARDWARE ERROR

on Firepower sensor LCD panel

BRKSEC-3455 77

Closing


Cisco Spark

Questions? Use Cisco Spark to communicate with the speaker after the session

1. Find this session in the Cisco Live Mobile App

2. Click “Join the Discussion”

3. Install Spark or go directly to the space

4. Enter messages/questions in the space

How

cs.co/ciscolivebot#BRKSEC-3455


• Please complete your Online Session Evaluations after each session

• Complete 4 Session Evaluations & the Overall Conference Evaluation (available from Thursday) to receive your Cisco Live T-shirt

• All surveys can be completed via the Cisco Live Mobile App or the Communication Stations

Don’t forget: Cisco Live sessions will be available for viewing on-demand after the event at www.ciscolive.com/global/on-demand-library/.

Complete Your Online Session Evaluation

http://ciscolive.com/Online

http://www.ciscolive.com/global/on-demand-library/


Continue Your Education

• Demos in the Cisco campus

• Walk-in Self-Paced Labs

• Tech Corner

• Meet the Engineer 1:1 meetings

• Related sessions

BRKSEC-3455 81

Thank you

Veronika Klauzova

Foster Lipkey

dissecting firepower-ftd · virtual fdm • allows users to ... config profile_preprocs: print all,...

Documents