quick start guide asa cluster on nexus

© 2013 Cisco and/or its affiliates. All rights reserved. 1

Quick Start Guide ASA Cluster on Nexus

Architecture & Solutions GroupUS Public Sector Advanced ServicesMark Stinnette, CCIE Data Center #39151

Date 20 August 2013Version 1.6.2


This presentation will provide end-to-end configurations mapped directly to commonly deployed data center architecture topologies. In this cookbook style; quick start guide; configurations are broken down in an animated step by step process to a complete end-to-end good clean configuration based on Cisco best practices and strong recommendations. Each QSG will contain set the stage content, technology component definitions, recommended best practices, and more importantly different scenario data center topologies mapped directly to complete end-to-end configurations. This QSG is geared for network engineers, network operators, and data center architects to allow them to quickly and effectively deploy these technologies in their data center infrastructure based on proven commonly deployed designs.

This Quick Start Guide (QSG) is a Cookbook style guide to Deploying Data Center technologies with end-to-end configurations for several commonly deployed architectures.


ASA Cluster ConfigurationCommonly Deployed Firewall Designs :: Standalone with Failover

• Cisco recommended • Commonly deployed & Typical firewall attachment model • ASA configured for port channels connected via vPC or vPC+• External and Internal traffic traverse same port channel to firewall• Insertion point at the Aggregation layer (Nexus 7000)• 10GE interfaces

• Altered ASA design topology • ASA configured for port channels connected via vPC or vPC+• Physical interface isolation for external and internal traffic

• External traffic traverse dedicated port channel to firewall• Internal traffic traverse dedicated port channel to firewall

• Insertion point at the Aggregation layer (Nexus 7000)• 10GE interfaces

• Altered ASA design topology • ASA VDC (Virtual Device Context) sandwich • ASA physically inline • ASA configured for port channels connected via vPC or vPC+• Physical interface isolation for external and internal traffic

• External traffic traverse dedicated port channel to firewall• Internal traffic traverse dedicated port channel to firewall

• Insertion point at the Aggregation layer (Nexus 7000)• External firewall port channel connected to Aggregation (VDC)• Internal firewall port channel connected to Sub-Aggregation

(VDC)• Uses more 10GE interfaces; less effective firewall bandwidth usage


• Cisco recommended :: ASA Cluster design • Scaling ASA appliances into one logical firewall within the DC architecture• Typical firewall cluster attachment model • ASA configured for port channels connected via vPC or vPC+• External and Internal traffic traverse same cluster data port channel to

firewall• Insertion point at the Aggregation layer (Nexus 7000)• 10GE interfaces• Cluster two or more (up to 8) ASA firewalls• Greatly increase the throughput of traffic (up to 100Gbps)• True active-active model; in multi-context mode every member interface

for all contexts are capable of forwarding every traffic flow

Same firewall Illustrated

Alternative View

Cluster up to 8 ASA firewalls ASA 5580 ASA 5585-X

ASA Cluster ConfigurationCommonly Deployed Firewall Designs :: Cluster Mode


ASA Cluster ConfigurationFirewall Logical Deployment Modes


Static RoutingDynamic Routing

No dynamic routing supported over vPC or vPC+

ASA Cluster ConfigurationFirewall Routing Considerations


Simple Tenant Container Single Tier model FW Context VRF VLAN mapping

High Security Use Cases N-Tier Application Segmentation Single FW Context instance Multiple VRFs to VLAN mappings

Enterprise-Class Data Center Service Provider / Cloud Zone Based Shared Multi-Tenant Context Single FW Context and VRF instance Multiple VLANs per Zone

ASA Cluster ConfigurationFirewall Logical Security Models :: Multi-Tenancy Infrastructure


Tenant Containers Private Public Shared Services DMZ N-Tier Application Segmentation

Rigorous Separation High Security Use Cases DoD / Federal Government Dedicated VRF per Tier Tenants mapped to unique firewall context

Service Provider / Cloud Enterprise-Class Data Center Zone Containers

Organization Departments Prod, Stage, Dev, Test Classification Types Application Type (Ent Apps, DB, BigData,

VDI) Zones mapped to firewall context Share the same Security Zone Container Optionally, virtual firewalls can be applied if

additional zoning is required within the containers (ie. VSG & ASA 1000v)

Unique Tenant Based ContainersZone Based Containers

ASA Cluster ConfigurationFirewall Logical Security Models :: Multi-Tenancy Infrastructure


The adaptation of an enterprise-wide security framework is a crucial part of the overall enterprise network architecture. Within the data center new application rollouts, virtualization, the adaptation of various cloud services and an increasingly transparent perimeter are creating radical shifts in the data center security requirements. The need for stackable scalable high capacity firewalls at the data center perimeter is becoming essential. Adaptive Security Appliance (ASA) clustering feature on the ASA family of firewalls satisfies such a requirement. The clustering feature allows for an efficient way to scale up the throughput of a group of ASAs, by having them all work in concert to pass connections as one logical ASA device.

Using up to 8 ASA appliances, the clustering feature allows the scaling of up to 100Gbps of aggregate throughput within the data center perimeter.

ASA Clustering provides the following benefits:• The ability to aggregate traffic to achieve higher throughput• Scaling the number of ASA appliances into one logical firewall within the Data Center architecture• True Active / Active model; when in multi-context mode every member for all contexts of the cluster are capable of

forwarding every traffic flow• Can force state-full flows to take more symmetrical path which improves predictability and session consistency• Can operate in either Layer 2 and Layer 3 modes• Supports single and multiple contexts (firewall virtualization)• (In Theory) Clustering can be implemented across different data centers over dark fibre as the means of

transport. This use case should be validated and supported in future releases • Cluster-wide statistics are provided to track resource usage• A single configuration is maintained across all units in the cluster using automatic configuration sync

ASA Cluster ConfigurationBenefits Overview


Cluster Data PlaneCluster Control Plane

cLACP Spanned Port ChannelNexus vPC

CL Master CL Slave CL Slave CL Slave

CL Master CL Slave CL Slave CL Slave

Same single vPC ID for all ASA units in the Cluster

ASA Cluster (n-node) Same Port Channel ID used across

all ASA units in the Cluster for the Data Links towards the Nexus Aggregation

vPC 10 vPC 20 vPC 30 vPC 40Unique vPC IDs used on the

Nexus Aggregation layer towards each ASA unit for the

CCL

vPC Domain (vPC or vPC+ supported)

Peer-Link

vPC 100

Po100 Po100 Po100 Po100

Po50 Po50 Po50 Po50Same Port Channel ID used across all ASA units in the Cluster for CCL towards the Nexus Aggregation layer

ASA Cluster ConfigurationTerminology & Components


Feature OverviewCluster Control Link (CCL) The CCL provides control plane information between the different cluster members. Also the flows are

redirected within the CCL. To configure the CCL, one configures local port channels with the same channel identifier on each firewall and connect them to separate vPCs on the corresponding Nexus7000s. All CCL links are part of same access VLAN.

Cluster Data Link The most important difference in implementing the cluster data plane is the configuration of a "spanned port channel (cLACP)" on the firewall. This is necessary because only one Port-Channel/vPC pair is used in the data plane. To provide channel consistency and seamless operation between both sides, it is necessary to configure a logical port-channel construct across all the members of the ASA cluster members. Data Link is a trunk port for all the inside and outside VLANs.

Spanned port channel (cLACP)

ASA uses a logical link aggregation construct called the Cluster Link Aggregation Control Protocol (cLACP). It is designed to extend standard LACP to multiple devices so that it can support span-cluster. EtherChannels need to be span across the cluster. cLACP allows link aggregation between one switch, or pair of switches, to multiple (more than two) ASAs in a cluster.

Local port channel(LACP)

Each ASA uses only two interfaces in a local port channel; meaning its not spanned or shared across the cluster. The local port-channel (vPC on the Nexus side) gives us local redundancy should we lose a single cluster control link.

LACP LACP (Link Aggregation Control Protocol) :: This is the protocol that the ASA runs to negotiate the ether channel to the adjacent switch. For clustering, the ASAs all share one instance of LACP, such that the adjacent switch considers the cluster of ASAs as one logical device.

Master The ASA Cluster elects a master unit that designates which unit responds to the cluster management address and which unit is used for configuration replication. All configuration is performed on the master unit. Hard set the master via the priority command.

Slave All other members in the cluster are slave units. Hard set the slaves accordingly via the priority command.

ASA Cluster ConfigurationAdditional Features, Terminology, & Components


Feature OverviewOwner Role Data path Packet Flow Through the Cluster

The unit that initially receives the connection. The owner maintains the TCP state and processes packets. A connection has only one owner.

The first ASA to receive traffic for a connection is designated as the owner

Director Role Data path Packet Flow Through the Cluster

The unit that handles owner lookup requests from forwarders and also maintains the connection state to serve as a backup if the owner fails. When the owner receives a new connection, it chooses a director based on a hash of the source/destination IP address and TCP ports, and sends a message to the director to register the new connection. If packets arrive at any unit other than the owner, the unit queries the director about which unit is the owner so it can forward the packets. A connection has only one director.

Forwarder Role Data path Packet Flow Through the Cluster

A unit that forwards packets to the owner. If a forwarder receives a packet for a connection it does not own, it queries the director for the owner, and then establishes a flow to the owner for any other packets it receives for this connection. The director can also be a forwarder. Note that if a forwarder receives the SYN-ACK packet, it can derive the owner directly from a SYN cookie in the packet, so it does not need to query the director (if you disable TCP sequence randomization, the SYN cookie is not used; a query to the director is required). For short-lived flows such as DNS and ICMP, instead of querying, the forwarder immediately sends the packet to the director, which then sends them to the owner. A connection can have multiple forwarders; the most efficient throughput is achieved by a good load-balancing method where there are no forwarders and all packets of a connection are received by the owner.



Feature OverviewCluster Connection(Owner Flow)

The actual connection flow that is passing the traffic. We can't know for sure which unit in the cluster will "own" the flow since whichever ASA receives the first packet in the flow will become the owner. Only TCP and UDP flows send logical flow updates to the stub flow (and possibly the director stub flow).

Cluster Connection(Forwarding Stub Flow)

If a unit receives a packet for a flow that it does not own, it will contact the director of that flow to learn which unit owns the flow. Once it knows this, it will create and maintain a forwarder flow, which it will then be used to forward any packets it receives on that connection directly to the owner, bypassing the director. Forwarder flows do not receive Link Updates (LUs) (since they're just forwarding the packets and don't care about state). Short lived flows such as DNS and ICMP will not have forwarder flows; the unit receiving the packets for those conns will simply forward them to the director, which will forward them to the owner, and the director will not reply back to the forwarder unit asking it to create a forwarder flow.

Cluster Connection(Backup Stub Flow)

Based on the flow's characteristics, all units can derive the Director unit for the flow. The director unit typically maintains the stub (or backup) flow, which can become the full flow in the case the flow's owner unit fails, and also be used to redirect units towards the flow's owner unit if they receive packets for the flow. Backup flows receive conn updates to keep them up-to-date in case the owner fails and the stub flow needs to become the full flow.

Cluster Connection(Stub or Backup Director Flow)

If the director chosen for the flow is also the owner (meaning the director received the first packet in the flow) then it can't be its own backup. Therefore a 'director backup' flow will be created, and a second hash table will be used to track this. Obviously this director backup flow will receive LUs, since it needs to be ready to take over if the director/owner fails.



Feature OverviewCluster Group Names the cluster and enters cluster configuration mode. The name must be an ASCII string from 1 to 38

characters. You can only configure one cluster group per unit. All members of the cluster must use the same name.

Local Unit Names this member of the cluster with a unique ASCII string from 1 to 38 characters. Each unit must have a unique name. A unit with a duplicated name will be not be allowed in the cluster.

Cluster Interface Specifies the cluster control link interface, preferably an Ether Channel. Specify an IP address; This interface cannot have a nameif configured. For each unit, specify a different IP address on the same network.

Console Replicate Enables console replication from slave units to the master unit. This feature is disabled by default. The ASA prints out some messages directly to the console for certain critical events. If you enable console replication, slave units send the console messages to the master unit so you only need to monitor one console port for the cluster.

Health Check ASA unit health monitoring and interface health monitoring. When you are adding new units to the cluster, and making topology changes on the ASA or the switch, you should disable this feature temporarily until the cluster is complete. You can re-enable this feature after cluster and topology changes are complete.

cLACP System Mac When using spanned Ether Channels, the ASA uses cLACP to negotiate the Ether Channel with the neighbor switch. ASAs in a cluster collaborate in cLACP negotiation so that they appear as a single (virtual) device to the switch. By default, the ASA uses priority 1, which is the highest priority.

Authentication Key Sets an authentication key for control traffic on the cluster control link. The shared secret is an ASCII string from 1 to 63 characters. The shared secret is used to generate the key. This command does not affect datapath traffic, including connection state update and forwarded packets, which are always sent in the clear.

Cluster Priority Sets the priority of this unit for master unit elections, between 1 and 100, where 1 is the highest priority.



ASA Characteristics 2-wide ASA cluster routed mode w/ static routing multi-context cluster spanned etherchannel mode

Nexus Characteristics 2-wide 7k Aggregation FabricPath vPC+ Static Routing & VRFs

Physical View – Connectivity Map

Each ASA has two 10GE interfaces connected to each respective Nexus 7K representing the data plane for the cluster. This is a spanned port-channel (recommended) across the ASA cluster in a single vPC. This is called the Cluster Data Link.

Each ASA has two 10GE interfaces in a local port channel (not spanned or shared across the cluster) called the Cluster Control Link (CCL). The CCL is the same on each ASA and will connect to the Nexus 7k via a unique vPC; since these are individual port channels and specific to each ASA.

ASA Cluster ConfigurationQuick Start Guide Assumptions


feature lacpfeature vpc

vlan 10-20, 2000 – 2999

spanning-tree pathcost method longspanning-tree port type edge bpduguard defaultspanning-tree port type edge bpdufilter defaultno spanning-tree loopguard default

spanning-tree vlan 10-20,2000-2999 priority 0spanning-tree pseudo-information vlan 10-20,2000-2999 root priority 4096 vlan 10-15,2000-2499 designated priority 8192 vlan 16-20,2500-2999 designated priority 16384

vpc domain 1 role priority 1 system-priority 4096 peer-keepalive destination [….] source [….] vrf management peer-switch peer-gateway auto-recovery auto-recovery reload-delay delay restore 30 ip arp synchronize

interface port-channel 2 switchport switchport mode trunk switchport trunk allowed vlan 10-20,2000-2999 spanning-tree port type network vpc peer-link interface e3/1 , e4/1 channel-group 2 force mode active


vlan 10-20, 2000 – 2999

spanning-tree pathcost method longspanning-tree port type edge bpduguard defaultspanning-tree port type edge bpdufilter defaultno spanning-tree loopguard default

spanning-tree vlan 10-20, 2000-2999 priority 0spanning-tree pseudo-information vlan 10-20,2000-2999 root priority 4096 vlan 10-15,2000-2499 designated priority 16384 vlan 16-20,2500-2999 designated priority 8192

vpc domain 1 role priority 2 system-priority 4096 peer-keepalive destination [….] source [….] vrf management peer-switch peer-gateway auto-recovery auto-recovery reload-delay delay restore 30 ip arp synchronize

interface port-channel 2 switchport switchport mode trunk switchport trunk allowed vlan 10-20,2000-2999 spanning-tree port type network vpc peer-link interface e3/1 , e4/1 channel-group 2 force mode active

ASA Characteristics 2-wide ASA cluster routed mode w/ static routing multi-context cluster spanned etherchannel

mode

See QSG :: vPC for more details …

ASA Cluster ConfigurationPrep for ASA Attachment :: vPC (Option)


feature lacpfeature vpcinstall feature-set fabricpathfeature-set fabricpath

vlan 10-20, 2000 – 2999 mode fabricpath

fabricpath switch-id 10

fabricpath domain default root-priority 255

spanning-tree pseudo-information vlan 10-20,2000-2999 root priority 0

vpc domain 1 role priority 1 system-priority 4096 peer-keepalive destination [….] source [….] vrf management peer-gateway auto-recovery auto-recovery reload-delay delay restore 30 ip arp synchronize fabricpath switch-id 1000

interface port-channel 2 switchport mode fabricpath vpc peer-link interface e3/1 , e4/1 channel-group 2 force mode active

feature lacpfeature vpcinstall feature-set fabricpathfeature-set fabricpath

vlan 10-20, 2000 – 2999 mode fabricpath

fabricpath switch-id 11

fabricpath domain default root-priority 254

spanning-tree pseudo-information vlan 10-20,2000-2999 root priority 0

vpc domain 1 role priority 2 system-priority 4096 peer-keepalive destination [….] source [….] vrf management peer-gateway auto-recovery auto-recovery reload-delay delay restore 30 ip arp synchronize fabricpath switch-id 1000

interface port-channel 2 switchport mode fabricpath vpc peer-link interface e3/1 , e4/1 channel-group 2 force mode activeSee QSG :: FabricPath for more details …

ASA Cluster ConfigurationPrep for ASA Attachment :: FabricPath vPC+ (Option)


mode


mode multiple

no firewall transparent

------------------------------------------------------

show activation-key

Serial Number: JMX1232L11M...Security Contexts : 10 perpetualCluster : Disabled perpetual…

activation-key ab42d738 a03b23fc 1bd3c87e d4d4c6d4 4e99ecbb

show activation-key

Serial Number: JMX1232L11M...Security Contexts : 10 perpetualCluster : Enabled perpetual…

port-channel load-balance src-dst ip-l4port

mode multiple

no firewall transparent

------------------------------------------------------

show activation-key

Serial Number: JMX1232L11M...Security Contexts : 10 perpetualCluster : Disabled perpetual…

activation-key ab42d738 a03b23fc 1bd3c87e d4d4c6d4 4e99ecbb

show activation-key

Serial Number: JMX1232L11M...Security Contexts : 10 perpetualCluster : Enabled perpetual…

port-channel load-balance src-dst ip-l4port

Step 1 :: enable multi-context modeStep 2 :: validate firewall status is routed Step 3 :: install | validate Cluster licenseStep 4 :: configure ECLB

Perform the configuration steps on the console port of each ASA.

Verify the firewall status as routed. If not routed, execute the no firewall transparent command. ciscoasa (config)# show firewall Firewall mode: Router

Enabling multi-context mode will force a reload; perform this on all the ASAs.

The clustering feature requires a specific license and code version 9.0.1 or greater. If you don’t have the proper license installed, refer to the “Managing Feature Licenses for Cisco ASA version 9.0” guide.

http://www.cisco.com/en/US/docs/security/asa/asa90/license/license_management/license.html

Traffic being load-balanced through ECLB :: it is important to choose a hash algorithm that is "symmetric," meaning that packets from both directions will have the same hash, and will be sent to the same ASA in the spanned Ether Channel. The hashing value selected should match between the aggregation switches and ASA, if possible.

ASA Cluster ConfigurationInitial Firewall Configuration & Verification Checks


mode


[system context]

cluster interface-mode spanned

interface Port-channel 40 description Clustering Interface port-channel load-balance src-dst ip-l4port

interface TenGigabitEthernet 0/8, 0/9 channel-group 40 mode active no nameif no security-level

cluster group ASA-CLUSTER key Cisc0! local-unit ASA-1 cluster-interface Port-channel40 ip 192.168.1.1 255.255.255.0 priority 1 console-replicate health-check holdtime 3 clacp system-mac auto system-priority 1 enable

[system context]



interface TenGigabitEthernet 0/8, 0/9 channel-group 40 mode active no nameif no security-level

cluster group ASA-CLUSTER key Cisc0! local-unit ASA-2 cluster-interface Port-channel40 ip 192.168.1.2 255.255.255.0 priority 2

enableStep 1 :: configure cluster interface typeStep 2 :: configure CCL local port channelsStep 3 :: enable clustering


interface port-channel 41 switchport switchport access vlan 10 spanning-tree port type edge no lacp graceful-convergence vpc 41




interface e1/1 channel-group 41 force mode active interface e1/2 channel-group 42 force mode active vlan 10 mode fabricpath name CLUSTER-CLL


master

vPC 41 vPC 42

ASA Cluster ConfigurationCluster Control Link


mode


[system context]



interface TE 0/8, 0/9 channel-group 40 mode active no nameif no security-level

cluster group ASA-CLUSTER key Cisc0! local-unit ASA-1 cluster-interface Port-channel40 ip 192.168.1.1 255.255.255.0 priority 1 console-replicate health-check holdtime 3 clacp system-mac auto system-priority 1 enable




Recommend you use a Ten Gigabit Ethernet interface for the cluster control link.

The recommended method is to use a spanned Ether Channel. When configured, if it detects any incompatibilities, it will clear them from the configuration and force a reload. This needs to be executed on each unit.

Each ASA communicates with each other across this common Vlan to form the cluster, update state information and pass data (when necessary).

The port channel configurations for 41, 42 on aggregation switch N7k-1 map to port-channel 40 on each ASA. The aggregation switch N7k-2 is configured the same with the only difference is that it physically connects to a different port (0/8) on each ASA. It is recommended to configure spanning-tree port type edge for the port-channels.

Port channel 40 is configured on each ASA and maps to 41, 42 on each N7k. The CCL interface configuration is not replicated from the master unit to slave units; however, you must use the same configuration on each unit. Ports te0/8 and te0/9 will be used for the CCL port-channel on each unit.

The ASA is actively negotiating LACP on the channel. This is another best practice; make sure all interfaces participating in channeling are actively using LACP. Also note there is no nameif or security-level configuration on the physical interfaces or the logical interface since this is being used for clustering control plane only.

All members of the cluster must share the same cluster group name and key if configured. The local-unit name, cluster-interface IP address and priority value needs to be unique for each unit in the cluster. The cluster master unit is determined by the priority setting, between 1 and 100, where 1 is the highest priority.

‘Enable’ command at the end of cluster configuration will start the cluster mode.

Step 1 :: configure cluster interface typeStep 2 :: configure CCL local port channelsStep 3 :: enable clustering

Console-replicate is an optional command that allows slave units to replicate console messages to the master. Since we spend most of our time on the master for configuration and troubleshooting purposes.

ASA Cluster ConfigurationCluster Control Link

NOTES


[system context]

mtu cluster 9216

jumbo-frame reservation

[system context]

mtu cluster 9216

jumbo-frame reservation

Step 1 :: enable mtu cluster [system context]Step 2 :: enable jumbo frame reservation [system context]Step 2 :: enable jumbo frame on the Nexus aggregation


vlan 10 mode fabricpath name CLUSTER-CLL

interface port-channel 41 switchport switchport access vlan 10 spanning-tree port type edge mtu 9216 no lacp graceful-convergence vpc 41


interface e1/1 channel-group 41 force mode active mtu 9216


vlan 10 mode fabricpath name CLUSTER-CLL





It is recommended to enable jumbo frame reservation and mtu cluster at least to1600 for the use with the cluster control link. When a packet is forwarded over cluster control link an additional trailer will be added, which could cause fragmentation. Set this to 9216 to match the system jumbo frame size configured on the N7k. Configure this on the master system context, save the config and then reboot the cluster.

A reboot is required to enable jumbo frames on the ASA.

ASA Cluster ConfigurationCluster Control Link & MTU


mode


[system context]

interface Management0/0

admin-context admincontext admin allocate-interface Management0/0 config-url disk0:/admin.cfg---------------------------------------------------------------

[admin context]

ip local pool mgmt 10.0.0.201-10.0.0.207 mask 255.255.255.0

interface Management0/0 management-only nameif mgmt security-level 100 ip address 10.0.0.200 255.255.255.0 cluster-pool mgmt

route mgmt 0.0.0.0 0.0.0.0 10.0.0.1 1---------------------------------------------------------------

[system context]

prompt hostname context cluster-unit

Step 1 :: allocate management interface [system context]Step 2 :: configure cluster management [admin context]Step 3 :: configure cluster host name prompt (optional) [system context]


In the system context allocate the management interface(0/0) to the admin context.

The management interface is configured with a primary IP address, along with a pool of addresses.

The primary management IP address always belongs to the current master unit, while the pool addresses are used to connect to each unit individually. Each unit, including the master gets a pool address assigned. You can connect to the master through either address, but if a failover should occur, the primary address will move to the new master. In the admin context configure the management IP addresses.

master

Display the pool IP addresses :: show ip local pool mgmt

ASA Cluster ConfigurationCluster Control Link & Management Access


mode


[system context]

interface Port-channel26 description Data Spanned Port-channel port-channel load-balance src-dst ip-l4port port-channel span-cluster vss-load-balance

interface TenGigabitEthernet 0/6 description Data Link to N7k-2 channel-group 26 mode active vss-id 1

interface TenGigabitEthernet 0/7 description Data Link to N7k-1 channel-group 26 mode active vss-id 2


interface port-channel 26 switchport switchport mode trunk switchport trunk allowed vlan 51, 2011-2012 spanning-tree port type edge trunk no lacp graceful-convergence vpc 26

interface e1/4, e1/5 lacp rate fast channel-group 26 force mode active

Step 1 :: configure Nexus aggregation port channelsStep 2 :: configure spanned data port channel

master


interface port-channel 26 switchport switchport mode trunk switchport trunk allowed vlan 51, 2011-2012 spanning-tree port type edge trunk no lacp graceful-convergence vpc 26

interface e1/4, e1/5 lacp rate fast channel-group 26 force mode active

It is recommended to configure the following for the best link aggregation and convergence ::

lacp rate fast no lacp graceful-convergence spanning-tree port type edge trunk

The N7k aggregation pair data port-channel is configured as a single vPC for all ASA units in the cluster. The vPC is configured as a trunk on the N7ks and as sub-interfaces on the ASA units.

The spanned data port-channel is configured in the ‘system context’. These port channels are shared across all ASA units and act as a single bundle. The N7k aggregation switches see this as a single port-channel, each having 4 interfaces configured.

The vss-id x command is used to identify the specific switch in the aggregation pair it connects to

The port-channel span-cluster vss-load-balance enables spanning.

Together these commands form the spanned Ether Channel. A spanned Ether Channel requires active LACP negotiation to be configured.

vPC 26

ASA Cluster ConfigurationCluster Data Link


mode


Logical Firewall Security Model

Now we have the network infrastructure built; lets configure a simple but yet flexible tenant container. Route summarization and static redistribution is used to advertise tenancy subnets into the Core or WAN Edge layer using OSPF. This will allow flexibility when adding additional server VLANs in any tenant without making any changes to static routes and routing at the aggregation layer. Since gateways for all VLANs within the VRF are at the aggregation layer, all interfaces are directly connected. No routing protocol is required to distribute routes within a given VRF.

ASA Context Characteristics Single Tiered Private Zone 1 outside VLAN 1 inside VLAN

Nexus Characteristics 1 VRF [internal private zone] 3 VLANs 3 HSRP Groups

[Outside, Inside, Server]

Security Container

ASA Cluster ConfigurationSimple Tenant Container


[system context]

interface Port-channel26 description Data Spanned Port-channel port-channel load-balance src-dst ip-l4port port-channel span-cluster vss-load-balance

interface TenGigabitEthernet 0/6 channel-group 26 mode active vss-id 1

interface TenGigabitEthernet 0/7 channel-group 26 mode active vss-id 2

interface Port-channel26.51 vlan 51



context Tenant_Zone_1 description Tenant Zone 1 FW Context allocate-interface Port-channel26.51 allocate-interface Port-channel26.2011 allocate-interface Port-channel26.2012 config-url disk0:/Tenant_Zone_1.cfg

Step 1 :: create sub-interfacesStep 2 :: create virtual firewall contextStep 3 :: allocate sub-interfaces to contextStep 4 :: configure context interfaces Step 5 :: configure context default routeStep 6 :: configure context static route(s) to servers vlans

master


The data port-channel is configured as sub-interfaces and allocated to the proper Tenant Zone context as required.

The context has a default route to the outside interface (N7k aggregation),

Followed by the security information which is configured for each context (sub-set shown here).

Port-channel26.51 is used for inband management (in this example)

[Tenant_Zone_1 context]

Hostname Tenant_Zone_1

interface Port-channel26.51 description Mgmt Vlan management-only nameif mgmt security-level 0 ip address 200.1.51.2 255.255.255.0

interface Port-channel26.2011 description Tenant Zone 1 OUTSIDE Vlan nameif outside security-level 10 ip address 200.1.1.11 255.255.255.0

interface Port-channel26.2012 description Tenant Zone 1 INSIDE Vlan nameif inside security-level 100 ip address 200.1.2.11 255.255.255.0

route outside 0.0.0.0 0.0.0.0 200.1.1.253 1route inside 200.1.3.0 255.255.255.0 200.1.2.253 1

while more specific routes are used to reach servers through the inside interface; those routes use the HSRP address as the gateway IP (N7k aggregation).

access-list inside-in extended permit ip any anyaccess-list outside-in extended permit ip any anyaccess-group outside-in in interface outsideaccess-group inside-in in interface inside



[N7k-1]

ip route 200.1.3.0/24 200.1.1.11

interface Vlan2011 description Tenant Zone 1 OUTSIDE Vlan mtu 9216 no ip redirects ip address 200.1.1.251/24 hsrp 1 ip 200.1.1.253

ip prefix-list static2ospfPfx seq 10 permit 200.0.0.0/10 le 24

route-map direct2ospf permit 10 match ip address prefix-list static2ospfPfx

router ospf 1 router-id [x.x.x.x] redistribute static route-map direct2ospf

Step 1 :: create firewall outside vlan SVI & HSRPStep 2 :: add static route for server vlan towards firewall context outside IP Step 3 :: redistribute server vlan into OSPF

[N7k-2]

ip route 200.1.3.0/24 200.1.1.11

interface Vlan2011 description Tenant Zone 1 OUTSIDE Vlan mtu 9216 no ip redirects ip address 200.1.1.252/24 hsrp 1 ip 200.1.1.253

ip prefix-list static2ospfPfx seq 10 permit 200.0.0.0/10 le 24

route-map direct2ospf permit 10 match ip address prefix-list static2ospfPfx

router ospf 1 router-id [x.x.x.x] redistribute static route-map direct2ospf


Note, the outside SVIs belong to the default global VRF. Nexus is already VRF aware and by default everything belongs to the default VRF.

Route summarization is used to advertise tenancy subnets into the Core / WAN Edge layer using OSPF. This allows adding of server VLANs in any tenancy without making any changes to static routes and routing at the aggregation layer.



[N7k-1]

vrf context Tenant_Zone_1 ip route 0.0.0.0/0 200.1.2.11

interface Vlan2012 description Tenant Zone 1 INSIDE Vlan mtu 9216 vrf member Tenant_Zone_1 no ip redirects ip address 200.1.2.251/24 hsrp 1 ip 200.1.2.253

interface Vlan2013 description Tenant Zone 1 SERVER Vlan mtu 9216 vrf member Tenant_Zone_1 no ip redirects ip address 200.1.3.251/24 hsrp 1 ip 200.1.3.253

Step 1 :: create tenant zone VRFStep 2 :: add default route to firewall context inside IP Step 3 :: create firewall inside vlan SVI & HSRPStep 4 :: create server vlan SVI & HSRP

[N7k-2]

vrf context Tenant_Zone_1 ip route 0.0.0.0/0 200.1.2.11

interface Vlan2012 description Tenant Zone 1 INSIDE Vlan mtu 9216 vrf member Tenant_Zone_1 no ip redirects ip address 200.1.2.252/24 hsrp 1 ip 200.1.2.253

interface Vlan2013 description Tenant Zone 1 SERVER Vlan mtu 9216 vrf member Tenant_Zone_1 no ip redirects ip address 200.1.3.252/24 hsrp 1 ip 200.1.3.253


The SVIs are configured to use HSRP. VLANs 2011 and 2012 represent the outside and inside interfaces of the ASA units for context Tenant_Zone_1. VLAN 2013 is used as a server VLAN. The inside VLANs are contained in a VRF to isolate the traffic and routing.

The AGG pair uses a default route in the VRF to route through the ASA cluster for outbound traffic.



[N7k-1]

vrf context Tenant_Zone_1 ip route 0.0.0.0/0 200.1.2.11 ip route 200.1.112.0/24 200.1.2.50

[N7k-2]

vrf context Tenant_Zone_1 ip route 0.0.0.0/0 200.1.2.11 ip route 200.1.112.0/24 200.1.2.50


Load Balancer vendor selection or configuration is outside scope of this document

[Tenant_Zone_1 context]

route outside 0.0.0.0 0.0.0.0 200.1.1.253 1route inside 200.1.3.0 255.255.0.0 200.1.2.253 1route inside 200.1.111.0 255.255.255.0 200.1.2.253 1

On the firewall context, add a specific route to reach the load-balancer through the inside interface; towards Nexus aggregation HSRP address. The route will use the alias IP address or floating IP address (similar to HSRP) on the load balancer.

On the Nexus aggregation, add a specific route to reach the load-balancer SNAT pool in the one-arm configuration; LB is the next hop.

On the load balancer add the default route towards the firewall’s inside interface and add a more specific route to the servers, towards the Nexus aggregation HSRP address.

Step 1 :: add firewall route to load balancer VIP [firewall context]Step 2 :: add route to load balancer SNAT address pool [Nexus aggregation]Step 3 :: add routes on load balancer

[Load Balancer virtual context]

interface [floating] ip address 200.1.2.50 /24

ip route 0.0.0.0/0 200.1.2.11Ip route 200.1.3.0/24 200.1.2.253



Here are some helpful commands executed in the ‘system context’ on the master unit:

• Shows the cluster status :: show cluster info• Shows cluster wide connection distribution :: show cluster info conn-distribution • Shows cluster wide packet distribution :: show cluster info packet-distribution

• Clear asp counters :: cluster exec clear asp drop • Show asp counters. Helpful to isolate drops :: cluster exec show asp drop • Shows the port channel summary on all units in the cluster :: cluster exec show port-channel summary • Shows all connections across the cluster. This command can show how traffic for a single flow arrives at different ASAs in

the cluster :: cluster exec show conn • Shows connection detail for a particular flow across all units in the cluster. Note, this needs to be executed in a context

that is handling the flow :: cluster exec show conn detail address [x.x.x.x]

• Show the unique MAC for the entire cluster that will be used for the LACP partner :: show lacp cluster system-id • Show the cluster system MAC (automatically generated) :: show lacp cluster system-mac

Commands executed in the ‘admin context’ on the master unit:

• Display the pool IP addresses :: show ip local pool mgmt

ASA Cluster ConfigurationShow Commands


• Clustering is best enabled in a specific, phased manner. To reduce the potential for errors, enable the CCL first and bring up the cluster before adding the remaining configuration. At a minimum, an active cluster control link network is required before you configure the units to join the cluster; this includes the upstream and downstream equipment port channels.

• When configuring clustering you need to select the cluster interface-mode first, as it will clear the existing configuration and force a reboot. It is recommended to use spanned Ether Channel.

• A console connection is always required to enable or disable clustering.

• Cluster control link bandwidth should match or exceed the highest available bandwidth of data interfaces on a single cluster unit.

• Recommend that you use Ten Gigabit Ethernet interfaces for the cluster control link, especially if there is high amount of centralized traffic or asymmetric traffic. If most traffic is centralized or asymmetric (undesirable) the cluster control link should have a higher bandwidth than data interface on each unit, because this traffic will have to be forwarded over cluster control link.

• Recommend that you use a port-channel for the CCL for additional resiliency. The port-channel configuration should use LACP mode active.

• The cluster control link should be in an isolated network and must not be a spanned Ether Channel. It needs to be configured on the aggregation switches as a unique port-channel for each unit in the cluster. ‘switchport access vlan [x]’

ASA Cluster ConfigurationStrong Recommendations and Key Notes


• It is recommended that spanning-tree port type edge or edge trunk is configured on the aggregation switch interfaces connecting to the cluster control and data interfaces. If this is not enabled, initial synchronization communication between ASA units in the cluster could fail and connections might be dropped.

• Using the same port channel load balancing hash algorithm between the ASA and Nexus 7000 (src-dst ip-l4port). Do not use the vlan keyword in the load-balance algorithm because it can cause unevenly distributed traffic to the ASAs in a cluster.

• Recommend that you do not specify the maximum and minimum links for a port-channel (The lacp max-bundle and port-channel min-bundle commands) on either the ASA or the switch.

• It is recommended that the spanned data port-channel is configured on the switch with no lacp graceful-convergence and lacp rate fast to achieve fast link aggregation and convergence.

• Recommend to use spanned Ether Channels (cluster interface-mode spanned) instead of individual interfaces because individual interfaces rely on routing protocols to load-balance traffic, and routing protocols often have slow convergence during a link failure.

• An IGP routing protocol peered with the ASA cluster does not provide the best convergence at the moment, static routes and Ether Channel Load Balancing (ECLB) is recommended to route and hash traffic to and from the ASA cluster. Note: dynamic routing is not supported over vPC or vPC+

• It is recommended to enable jumbo frame reservation and mtu cluster 1600 for use with the cluster control link (CCL). When a packet is forwarded over cluster control link an additional trailer will be added, which could cause fragmentation.



• For the management interface, we recommend using one of the dedicated management interfaces (m0/0 or m0/1). This should be configured to use an isolated network apart from the CCL or data interface configuration.

• In spanned Ether Channel mode, if you configure the management interface as an individual interface, you cannot enable dynamic routing for the management interface. You must use a static route.

• Recommend that you manually force an ASA unit to be the designated master and the other units as slaves via the priority command under the cluster group configuration.

• In single context mode, it is strongly recommended to configure static MAC addresses for a spanned Ether Channel, so that the MAC address does not change when the current master unit leaves the cluster. Manually configured MAC addresses will always stay with the master unit.

• In multiple context mode, if you share an interface between contexts, auto-generation of MAC addresses is enabled by default. You should verify this to avoid any potential issues. The following command ‘mac-address auto prefix 1’ in the configuration is used to auto-generate MAC addresses

• Note :: In spanned Ether Channel mode, if you configure the management interface as an individual interface, you cannot enable dynamic routing for the management interface. You must use a static route.

• Note :: you enable clustering when you enter the ’enable’ command under the cluster group configuration. If you disable clustering, all data interfaces are shut down, and only the management interface is active.

• A Cluster license is required on each unit. For other feature licenses, cluster units do not require the same license on each unit. If you have feature licenses on multiple units, they combine into a single running ASA cluster license. Note, each unit must have the same encryption license when in cluster mode.



• Recommended in principle to first maximize the number of active ports in the channel, and secondly keep the number of active primary ports and the number of active secondary ports in balance. Having an even number of ASA units in the clusters will allow traffic to balance evenly.

Note that when an odd number unit joins the cluster, traffic is not balanced evenly between all units. Link or device failure is handled with the same principle; you may end up with a less-than-perfect load balancing situation.

• Recommend to use the health check feature; which is configured under the cluster group configuration and the default holdtime is 3 seconds. After you add all the slave units, and the cluster topology is stable, re-enable the cluster health check feature, which includes unit health monitoring and interface health monitoring. Keepalive messages between members determine member health. If a unit does not receive any keepalive messages from a peer unit within the holdtime period, the peer unit is considered unresponsive or dead.

• When any topology changes occur (such as adding or removing a data interface, enabling or disabling an interface on the ASA or the switch, or adding an additional switch to form a vPC) you should disable the health check feature. When the topology change is complete, and the configuration change is synced to all units, you can re-enable the health check feature.

• When the firewall is deployed in transparent mode (vlan translation between inside and outside vlans that belong to same bridge-group with associated BVI interface) all cluster configuration recommendations remain the same; but an additional strong recommendation is to filter STP BPDU forwarding using an access-list on the inside and outside interfaces when the ASA Cluster is connected to a vPC or vPC+ domain on the Nexus platform.

access-list 1 ethertype deny bpduaccess-group 1 in interface insideaccess-group 1 in interface outside



External (public)ASA Clustering within VMDC Architecture http://www.cisco.com/en/US/docs/solutions/Enterprise/Data_Center/VMDC/ASA_Cluster/ASA_Cluster.html

VMDC (Virtual Multi-Service Data Center) 3.0.1 Implementation Guide http://www.cisco.com/en/US/partner/docs/solutions/Enterprise/Data_Center/VMDC/3.0.1/IG/VMDC301_IG1.html

ASA 5500 Configuration Guideshttp://www.cisco.com/en/US/partner/products/ps6120/products_installation_and_configuration_guides_list.htmlConfigure a Cluster of ASAs (version 9.1 code)http://www.cisco.com/en/US/partner/docs/security/asa/asa91/configuration/general/ha_cluster.html Nexus 7000 Configuration Guideshttp://www.cisco.com/en/US/products/ps9402/products_installation_and_configuration_guides_list.html

ASA Cluster ConfigurationAdditional Resources & Further Reading

Great External Resources

http://www.cisco.com/en/US/docs/solutions/Enterprise/Data_Center/VMDC/ASA_Cluster/ASA_Cluster.html

http://www.cisco.com/en/US/docs/solutions/Enterprise/Data_Center/VMDC/ASA_Cluster/ASA_Cluster.html

http://www.cisco.com/en/US/partner/docs/solutions/Enterprise/Data_Center/VMDC/3.0.1/IG/VMDC301_IG1.html

http://www.cisco.com/en/US/partner/products/ps6120/products_installation_and_configuration_guides_list.html

http://www.cisco.com/en/US/partner/products/ps6120/products_installation_and_configuration_guides_list.html

http://www.cisco.com/en/US/partner/docs/security/asa/asa91/configuration/general/ha_cluster.html



http://www.cisco.com/en/US/products/ps9402/products_installation_and_configuration_guides_list.html

http://www.cisco.com/en/US/products/ps9402/products_installation_and_configuration_guides_list.html


Quick Start Guide :: Virtual Port Channel (vPC)https://communities.cisco.com/docs/DOC-35728

Quick Start Guide :: FabricPathhttps://communities.cisco.com/docs/DOC-35725l

ASA Cluster ConfigurationAdditional Resources & Further Reading

https://communities.cisco.com/docs/DOC-35728



http://www.cisco.com/en/US/partner/docs/solutions/Enterprise/Data_Center/VMDC/3.0.1/IG/VMDC301_IG1.html

quick start guide asa cluster on nexus

Documents

external firewall port

cluster data port channel

port channels

inline asa

asa firewallsgreatly

vpc external

aggregation layer nexus

data center technologies