Chapter 6Configuring, Monitoring &

Troubleshooting IPsec

6.1 Overview of IPsecBenefits of IPsecRecommended Uses of IPsecTools Used to Configure IPsecWhat are Connection Security Rules ?

Benefits of IPsecIPsec – suite of protocols that allows secure,

encrypted communication between 2 computers over an unsecured network

2 goals; to protect IP packet & to defend against network attacks

IPsec secures network traffic by using encryption & data signing

An IPsec policy defines the type of traffic that Ipsec examines, how that traffic is secured & encrypted, and how IPsec peers are authenticated

Recommended Uses of IPsecAuthenticating & encrypting host-to-host

trafficAuthenticating & encrypting traffic to serversLayer 2 Tunneling Protocol (L2TP)/IPsec for

VPN connectionsSite-to-site (gateway-to-gateway) tunnelingEnforcing logical networks (server/domain

isolation)

Tools Used to Configure IPsecWindows Firewall with Advanced Security

MMC (used for Windows Server 2008 & Windows Vista)

IP Security Policy MMC (used for mixed environments & to configure policies that apply to all Windows versions)

Netsh command-line tool

What are Connection Security Rules?Connection security rules involve:- Authenticating 2 computers before they begin

communications- Securing information being sent between 2 computers- Using key exchange, authentication, data integrity &

data encryption (optionally)How firewall rules & connection rules are related:- Firewall rules allow traffic through, but do not secure

that traffic- Connection security rules can secure the traffic, but

creating a connection security rule does not allow traffic through the firewall

6.2 Configuring Connection Security RulesChoosing a Connection Security Rule TypeWhat are Endpoints?Choosing Authentication RequirementsAuthentication MethodsDetermining a Usage Profile

Choosing a Connection Security Rule TypeRule Type Description

Isolation Restricts connections based on authentication criteria that you define

Authentication Exemption

•Exempts specific computers, or a group or range of IP addresses, from being required to authenticate•Grants access to those infrastructure computers with which this computer must communicate before authentication occurs

Server-to-server Authenticates 2 specific computers, 2 groups of computers, 2 subnets, or specific computer & a group of computers or subnet

Tunnel Provides secure communications between 2 peer computers through tunnel endpoints (VPN or L2TP IPsec tunnels)

Custom Enables you to create a rule with special settings

What are Endpoints?Computer endpoints are the computers or the

group of computers that form peers for the connection

IPsec tunnel mode protects an entire IP packet by treating it as an AH or ESP payload

ESP encrypts packets and applies a new unencryptes header to facilitate routing

ESP function in 2 modes:1. Transport mode2. Tunnel mode

IP HDR Data

IP HDR ESP HDR

Encrypted Data

ESP TRLR

ESP Auth

IP HDR Data

ESP HDR

ESP TRLR

ESP Auth

New IP HDR

Encrypted IP Packet

ESP Transport Mode

ESP Tunnel Mode

Choosing Authentication RequirementsOption Description

Request Authentication for inbound and outbound connections

Ask that all inbound/outbound traffic be authenticated, but allow the connection if authentication fails

Require authentication for inbound connections and request authentication for outbound connections

•Require inbound be authenticated or it will be blocked•Outbound can be authentication fails

Require authentication for inbound and outbound connections

Require that all inbound/outbound traffic be authenticated or the traffic will be blocked

Authentication MethodsMethod Key Points

Default Use the authentication method configured on the IPsec Settings tab

Computer & User (Kerberos V5)

You can request or require both the user & computer authenticate before communications can continue; domain membership required

Computer (Kerberos V5) Request or require the computer to authenticate using Kerberos V5

User (Kerberos v5) Request or require the user to authenticate using Kerberos V5; domain membership required

Computer certificate •Request or require a valid computer certificate, requires at least one CA•Only accept health certificates: request or require a valid health certificate to authenticate, requires IPsec NAP

Advanced Configure any available method; you can specify methods for First & Second Authentication

Determining a Usage ProfileSecurity settings can change dynamically with the

network location typeWindows supports 3 network types :- Domain: selected when the computer is a domain

member- Private: networks trusted by the user (home or small

office network)- Public: default for newly detected networks, usually

the most restrictive settings are assigned because of the security risks present on public networks

The network location type is most useful on portable computers which are likely to move from network to network

6.3 Configuring IPsec NAP EnforcementIPsec Enforcement for Logical NetworksIPsec NAP Enforcement ProcessesRequirements to Deploy IPsec NAP

Enforcement

IPsec Enforcement for Logical Networks

SHAs NAP agent NAP ECs

HRAVPN802.1xDHCPNPS proxy

NAP administration serverNetwork policiesNAP health policiesConnection request policiesSHVs

SHAsNAP agentNAP ECs

Certificate servicesEmail serversNAP policy servers

Secure servers

Compliant NAP client

Secure NetworkBoundary Network

Restrictednetwork

NAP enforcement servers

Remediation servers

Non-compliantNAP client

Non-NAPCapable client

NPS servers

IPsec NAP Enforcement Processes

VPN Server Active

Directory

IEEE 802.1x Devices

Health Registration Authority

NAP Health Policy Server

DHCP Server

Remediation Server

NAP Client with limited access

Perimeter Network

Internet

Intranet

Restricted network

IPsec NAP Enforcement includes:

• Policy validation• NAP enforcement• Network restriction• Remediation• Ongoing monitoring of compliance

Requirements to Deploy IPsec NAP EnforcementActive DirectoryActive Directory Certificate ServicesNetwork Policy ServerHealth Registration Authority

6.4 Monitoring IPsec ActivityTools used to Monitor IPsecUsing IP Security Monitor to Monitor IpsecUsing Windows Firewall with Advanced

Security to Monitor IPsec

Tools Used to Monitor IPsecTool Key Points

IP Security Monitor

• Used in Windows XP and higher

• MMC snap-in

• Administrators can monitor local and remote IPsec policy usage

IPsecmon

• Only available in Windows 2000

• Command-line tool

• Reduced level of information available for troubleshooting

Windows Firewall with Advanced Security MMC

New in Windows Vista and Windows Server 2008

Detailed IKE tracing using Netsh

• Trace file found in: systemroot\debug\oakley.log

• Enabled in Windows XP and Windows 2000 through Registry modification

Using IP Security Monitor to Monitor IPsec

Options for using the IP Security Monitor:• Modify IPsec data refresh interval to update information in the

console at a set interval• Allow DNS name resolution for IP addresses to provide additional

information about computers connecting with IPsec• Computers can monitored remotely:

• To enable remote management editing, the HKLM\system\currentcontrolset\services\policyagent keymust have a value of 1

• To Discover the Active security policy on a computer, examine the Active Policy Node in the IP Security Monitoring MMC

• Main Mode Monitoring monitors initial IKE and SA:• Information about the Internet Key Exchange

• Quick Mode Monitoring monitors subsequent key exchanges related to IPsec:

• Information about the IPsec driver

Using Windows Firewall with Advanced Security to Monitor IPsec

• Use the Connection Security Rules and Security Associations nodes to monitor IPsec connections

• The Connection Security Rules and Security Associations nodes will not monitor policies defined in the IP Security Policy snap-in

• Items that can be monitored include: • Security Associations• Main Mode• Quick Mode

The Windows Firewall in Windows Vista and Windows Server 2008 incorporates IPsecThe Windows Firewall in Windows Vista and Windows Server 2008 incorporates IPsec

6.5 Troubleshooting IPsecIPsec Troubleshooting ProcessTroubleshooting Internet Key Exchange

(IKE)Troubleshooting IKE Negotiation Events

IPsec Troubleshooting Process

Stop the IPsec Policy Agent and use the ping command to verify communications

Verify firewall settings

Start the IPsec Policy Agent and use IP Security Monitor to determine if a security association exists

Verify that the policies are assigned

Review the policies and ensure they are compatible

11

22

33

44

55

Use IP Security Monitor to ensure that any changes are applied66

Troubleshooting IKE

Identify connectivity issues related with IPsec and IKEüü

Identify firewall and port issuesüü

View the Oakley.log file for potential issuesüü

Determine Main mode exchange issuesüü

Troubleshooting IKE Negotiation EventsCommon Security Event log codes:

• Success:• 541 - IKE Main Mode or Quick Mode established• 542 - IKE Quick Mode was deleted• 543 - IKE Main Mode was deleted 

• Information Log Entries:• Largely pertains to monitoring for denial of service attacks • There might not be any errors but resources will

run low, which affects performance for legitimate clients• Quick Mode audit failures are denoted with 547 error message

End of Chapter 6