idp series policy design and optimization - juniper …€¦ · juniper networks idp series...

IMPLEMENTATION GUIDE

Copyright © 2009, Juniper Networks, Inc. 1

IDP SERIES POLICY DESIGN AND OPTIMIZATION

Although Juniper Networks has attempted to provide accurate information in this guide, Juniper Networks does not warrant or guarantee the accuracy of the information provided herein. Third party product descriptions and related technical details provided in this document are for information purposes only and such products are not supported by Juniper Networks. All information provided in this guide is provided “as is”, with all faults, and without warranty of any kind, either expressed or implied or statutory. Juniper Networks and its suppliers hereby disclaim all warranties related to this guide and the information contained herein, whether expressed or implied of statutory including, without limitation, those of merchantability, fitness for a particular purpose and noninfringement, or arising from a course of dealing, usage, or trade practice.

2 Copyright © 2009, Juniper Networks, Inc.

IMPLEMENTATION GUIDE - IDP Series Policy Design and Optimization

Table of Contents

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

Design Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

Solution-Type Design Guidance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

General Topology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

Software and Hardware Versions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7

Solution Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7

Components of the Policy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7

Designing and Implementing a Policy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7

Defining Network and Host Objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7

Defining a Policy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

Steps to Implement a Recommended Policy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

Defining Actions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10

Defining Notifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

Customizing the Policy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

Server Protection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

Create Granular Notification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

Predefined and Custom Dynamic Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

Terminal Rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

Exempt Rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

False Positives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

Performance Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .16

Optimize Traffic Filtering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .16

Client-to-Server/Server-to-Client . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .16

Notifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .16

Application Identification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .16

Fine-tuning and Monitoring the Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .16

Security Profiler . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

Security Explorer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

SCTOP Command-Line Utility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .18

Appendix A: Resources for Ongoing Maintenance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .18

Daily Signature Updates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .18

Detector Updates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .18

Known Issues Update . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .19

About Juniper Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .19



Table of Figures

Figure 1: IDP Series policy using the All_With_Logging template . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

Figure 2: Sample IDP Series deployment topology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

Figure 3: Defining a Web server as a host in NSM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

Figure 4: Predefined IDP Series policy templates available within NSM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

Figure 5: Creating a new security policy based on the recommended policy template . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

Figure 6: IDP Series rules created in the recommended policy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10

Figure 7: Detailed description of a sample signature . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

Figure 8: Rules protecting internal servers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

Figure 9: Creating granular notification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

Figure 10: Listing of predefined attack group categories . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

Figure 11: Defining a custom dynamic group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

Figure 12: Exempt rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15



Introduction

As the public Internet is increasingly infiltrated with rogue network traffic and threats, enterprises face the

challenges of providing secure network access while protecting their users, applications, and services from attack.

Juniper Networks® IDP Series Intrusion Detection and Prevention Appliances provide network administrators with a

powerful tool to monitor and prevent network threats—both at the protocol and application levels—by employing a

comprehensive, signature-based, attack database used in conjunction with a rule-based policy. As malware such as

worms and trojans often target vulnerabilities in protocols, the IDP Series also uses a feature called protocol anomaly

detection in conjunction with stateful signatures to inspect and detect threats before they reach the intended target.

In addition, IDP Series appliances can regulate and block unwanted peer-to-peer applications or instant messaging to

maintain compliance with corporate network use policies.

As network requirements and services are very diverse and the threats attacking them are continually evolving,

configuring intrusion prevention system (IPS) security policy to provide optimal coverage is a challenging task for

administrators. The IDP Series provides administrators with a highly flexible and configurable security policy and threat

coverage toolset. Often this level of flexibility comes with complexity and confusion around how IDP Series appliances

process rules and mitigate network threats. Once understood, the IDP Series provides administrators with a robust

number of policy management features to design the security rulebase, manage the attack objects applied, and

designate an appropriate action in response.

This document is designed to help security administrators, network operations engineers, and implementation partners

design and implement effective IDP Series security policies using the recommended policy, as well as customize and

optimize the configuration using additional features available within the IDP Series platform.

Scope

Network security administrators are continually challenged with a barrage of threats against their networks. Operating

system and application vulnerabilities are found on a daily basis. Malware such as botnets, worms, trojans, and spyware

infiltrate and steal valuable resources. Enterprises, large or small, must thwart these attacks to protect their intellectual

property. Multiple directions of threat, both internal and external, further complicate this complex task. Additionally,

unwanted peer-to-peer and chat applications constrain network bandwidth and reduce worker productivity.

In addition, increased compliance regulations and privacy laws require protections to ensure identity protection and

safeguarding of financial data. Health organizations and financial institutions are a few examples of organizations

required by federal regulations to protect consumer data from network breaches. Sensitive information such as

medical records, financial transactions, social security numbers, and credit card numbers must be protected against

unauthorized harvesting by malware. Regulatory compliance to standards such as the Health Insurance Portability and

Accountability Act (HIPAA), Peripheral Component Interconnect (PCI), and the Sarbanes-Oxley Act is essential.

Juniper Networks IDP Series Intrusion Detection and Protection Appliances provide a strong platform backed by the

Juniper Networks Security team to block network threats and ensure corporate network policy. The IDP Series offering

is a comprehensive tool, capable of providing key elements of compliance requirements. Both the standalone and

integrated Juniper Networks ISG Series Integrated Security Gateways and IDP Series are deployed in a range of small

to large enterprises, across multiple verticals. Additionally, service providers are deploying high-throughput versions of

IDP Series products. Data center applications require IPS protection to manage increasing threat sophistication as well

as rising regulatory pressures. The IDP Series provides a critical, focused defense behind a network firewall to block

network attacks and manage application-level traffic. Optimization of this layer is highly desirable, along with accurate

tuning and elimination of false positives.

This document is designed for security administrators, network operations engineers, and implementation partners. It

provides a design solution to build and optimize an effective security policy by implementing several newly developed

features in the IDP Series: recommended policy and recommended actions. The solution also discusses approaches

and features to customize IDP Series policy. Using these guidelines, an administrator can:

• Provide a thorough level of threat coverage

• Simplify ongoing policy management

• Minimize exposure to false positives

• Optimize IDP Series performance



Design Considerations

While Juniper Networks IDP Series Intrusion Detection and Prevention Appliances provide an excellent defense against

today’s threats, administrators are often perplexed by the design of IPS policy and the selection of attack signatures to

defend against.

Often, administrators new to IPS will configure an IDP Series policy of type Any/Any/Any using the All_With_Logging

predefined template, hoping to get the full effect of the complete IDP Series signature database to protect their

networks. This will create a policy that matches Any source, Any destination, and inspects against Any attack signature.

This configuration, shown in the following screen capture, will result in an IDP Series policy that will inspect every

packet against the entire signature database—creating too many alerts and false positives, and resulting in

poor performance.

Figure 1: IDP Series policy using the All_With_Logging template

This document presents best practices to design an effective security policy using the following concepts and

features available within the IDP Series and its management software platform, Juniper Networks Network and

Security Manager.

• Begin with a network topology with defined protected segments

• List and define network segments, host, and server address objects within the NSM UI

• Create a new policy using the predefined “recommended policy” as a template

• Create specific rules for servers

• Review recommended actions

• Create and manage dynamic attack groups

• Create exempt rules

• Utilize Application Identification

• Leverage the Enterprise Security Profiler

• Correlate traffic with the Security Explorer



Solution-Type Design Guidance

General Topology

The following topology is a typical IPS deployment at an enterprise where the IDP Series resides on the network edge

behind a firewall. The IDP Series in this position is inline with network traffic and can protect the servers (SMTP, WWW,

and DNS) as well as the additional clients and resources on the LAN. This sample topology will be used as a reference

for creating an IDP Series policy.

Figure 2: Sample IDP Series deployment topology

Software and Hardware Versions

This document is based on IDP Series release 4.1r2 along with NSM 2007.3r2.

The features and configuration samples discussed in this document were tested on a standalone Juniper Networks

IDP1100 Intrusion Detection and Prevention Appliance, though the principles apply to any standalone or integrated ISG

Series with IDP Series security module. Differences between standalone IDP Series products and integrated ISG Series/

IDP Series products are noted where applicable.

Border Router

NetScreen-5200Firewall

IDP Series

NetScreen-5200

Network andSecurity Manager

SMTP WWW DNS

DMZ

INTERNET



Implementation

Solution Description

This document details a recommended approach to designing and customizing an effective IPS security policy by

specifying the usage of a number of key IDP Series features.

The IPS security policy in its entirety consists of the main IDP Series rulebase and five other rulebases: Exempt,

Backdoor, SYN Protector, Traffic Anomalies, and Network Honeypot. This document will focus on designing

and optimizing the main IDP Series rulebase and will also discuss the exempt rulebase. Juniper Networks NSM

Administrator’s Guide discusses the other rulebases in great detail.

Components of the Policy

The IDP Series main rulebase policy consists of the following essential parameters:

• Rule Sequence—The IDP Series policy processes traffic through a numbered sequence of rules, allowing for

prioritization of earlier rules over later rules.

• Address Objects—Before traffic inspection, the IDP Series filters traffic based on source and destination IP

addresses.

• Service—IDP Series appliances can filter inspection based on a transport layer service ((TCP, UDP, remote procedure

call (RPC), Internet Control Message Protocol (ICMP)) or port.

• Attack Objects—These form the basis of the stateful traffic inspection. In addition to REGEX-based pattern-

matching signatures, attack objects also consist of protocol anomalies that detect traffic that deviates from a

standard protocol. Attack objects are grouped into five severity levels (Critical, Major, Minor, Warning, and Info) as

well as many categories of traffic.

• Action—This defines the way IDP Series appliances handle traffic when an attack object is matched. There are a

total of nine actions available, which are discussed later in this guide.

• Notification—Each rule can have multiple notification responses, including recording a log event, various forms of

alerts, and packet logging.

This document will cover each of the aforementioned parameters while providing recommendations and examples to

help beginning IDP Series administrators create an effective policy suitable for their environment.

Designing and Implementing a Policy

Defining Network and Host Objects

The first step in creating a security policy is to identify the network segments and resources being protected. A detailed

network topology should be consulted to note subnets and work groups so that rules can be customized to permit and

deny network services. Servers should also be identified at this point and grouped by function so that attack objects

can be appropriately applied.

Within NSM, networks and hosts are added into the Object Manager as Address Objects. Once individual entries are

created, objects can be added to address groups to simplify rule creation.

The following figure defines a Web Server with an IP address of 10.1.2.101. After multiple hosts are created, hosts

requiring similar protection can be added to Groups.



Figure 3: Defining a Web server as a host in NSM

Defining a Policy

Policy Templates

After defining Address Objects, one can begin defining the rules governing the policy. To simplify this, NSM has several

predefined IDP Series policies from which to choose:

Figure 4: Predefined IDP Series policy templates available within NSM

Some of these templates implement all attack objects. These are useful for product demos but are not practical

in production deployments as they are resource intensive. Some of the templates are specific to certain network

topologies: demilitarized zone (DMZ) environment, Domain Name System (DNS) server, FTP server, Web server, and

file server.

Recommended Policy

Customers are advised to start with the recommended policy as a template and customize additional rules with

specific networks, hosts, and application/protocol attacks that are significant for their network.

The recommended policy is a specifically designed set of rules and attack objects grouped into a policy by the Juniper Networks

Security team, which provides protection against the most relevant threats known to be present and proliferating through the

Internet. This policy provides a good starting point for protection, while allowing administrators to subsequently analyze traffic

traversing IDP Series appliances to further optimize the policy and best address their specific network policy requirements. This

approach will yield optimal IDP Series performance, while providing proper and appropriate coverage, and minimizing false

positive alerts and disruptions to legitimate traffic.

The attack groups contained within the recommended policy are updated as new vulnerabilities and threats are

found, so that the recommended policy or policies using the recommended attack groups have the latest signatures

automatically. The recommended policy also includes the attack objects protecting against Microsoft’s latest

vulnerabilities, typically providing zero-day coverage of these threats.

Updating the NSM Attack Database and applying it to IDP Series appliances will update the recommended policy with

the latest attack objects. The update procedure is further discussed in Appendix 1: Ongoing Maintenance.



Steps to Implement a Recommended Policy

To create a new Policy based upon the recommended policy, select Security Policies within Policy Manager and click

on the (+) to create a new Security Policy. After naming the policy and specifying IDP Series as the device type, select

Use Predefined IDP Policy as Template. Choose recommended (predefined). Finally, assign the new policy to an IDP

Series appliance.

Figure 5: Creating a new security policy based on the recommended policy template

The recommended policy consists of nine rules in the main rulebase, providing protection against important TCP, ICMP,

HTTP, Simple Mail Transfer Protocol (SMTP), DNS, FTP, POP3, Internet Message Access Protocol (IMAP) attacks as

well as common Internet malware. The following figure shows the default recommended policy with the nine rules

applied to all traffic traversing IDP Series appliances.



Figure 6: IDP Series rules created in the recommended policy

Defining Actions

When traffic matches an attack object, the IDP Series provides a number of possible actions as a response: No action,

Ignore remainder of connection, Drop Packet, Drop Connection, Close Client, Close Server, and Differentiated Services

(DiffServ) Marking. The Close actions will send reset (RST) packets for TCP traffic, while Drop actions will drop a

packet or connection without an RST packet. Depending on the nature of the attack and network service, it can be

advantageous to choose one action over another.

Recommended Actions

To simplify action selection, the Juniper Networks Security team has implemented a predefined recommended action

for signatures in the attack object database. These recommended actions are used by default in the recommended

policy and are available in standalone IDP Series 4.1 and Juniper Networks ScreenOS 6.0 for integrated ISG Series/

IDP Series products. The recommended action for a specific signature can be viewed by selecting Attack Objects > IDP

Objects > choose the specific signature > right-click > View.

The Juniper Networks Security team’s predefined recommended action provides the most appropriate response for the

threat. Traffic matching the most significant attack objects is dropped, while minor attack objects are marked for no

action. Recommended actions are assigned to predefined attack objects based on attack severity:



The recommended checkmark and recommended action fields are visible as well, as the detailed description of the

signature as shown in the following figure:

Figure 7: Detailed description of a sample signature

Defining Notifications

Once an action is taken, the IDP Series can be configured to provide notification of the event in multiple forms: logging,

email alerts, SNMP alerts, syslog alerts, trigger a script, and packet logging are a few of the options available. The

default notification in the recommended policy is to log the attacks.

Customizing the Policy

While the recommended policy provides a good baseline policy, an effective policy will have to be tailored to the

specific network, taking into account specific applications and servers used and any corporate or regulatory policies

needing enforcement.

Server Protection

The first step of customization is to create additional rules to protect internal servers. The rules apply additional server-

specific attack objects to internal servers. Using the simplified network topology presented earlier as an example,

additional rules are created to protect identified Web, SMTP, and DNS servers. While the recommended policy provides

basic protection of these services applicable to all traffic, more complete coverage for servers is recommended. One

may add predefined attack groups based on category and severity within the attack column.



In the following example, three new rules have been added to the recommended policy. Critical, Major, and Minor

Attack Objects are applied in rules specific to traffic destined for the mail, Web, and DNS servers. Additional database,

backup, or media servers should have similar rules applied.

Figure 8: Rules protecting internal servers

Create Granular Notification

In the previous example, notification action on all servers protecting rules is set as “Logging” despite having attacks

of differing severities. In a typical deployment, critical attacks will require alert notification so that quick corrective

action can be taken. To make the notification more granular, split the critical attacks into their own rules while adding

additional notification. Notification options including email, SNMP, syslog, and packet capture are all available by right-

clicking the Notification field.

In the following example, critical attacks on the Web servers will generate alert notifications, while minor-level SMTP,

Web, and DNS attacks have been configured to no notification. No notification may be desired if the network traffic

generates too much log data.

Figure 9: Creating granular notification

Predefined and Custom Dynamic Groups

To further customize the IDP Series rulebase, administrators need to apply custom rules built with relevant attack

objects for applications specific to their network. As the IDP Series attack object database is very comprehensive, it

is impractical to comb through it looking for relevant attacks. NSM addresses this by subdividing the attack database

into multiple predefined attack groups. Administrators can create very specific rules by using attack groups with

application- and protocol-specific categories.



Figure 10: Listing of predefined attack group categories

Administrators may find the need to customize attack groups based on application, traffic direction, or severity. NSM

allows this by using custom attack groups. Custom attack groups can be created by filtering attacks based on the

following criteria:

• Product type—application, protocol, operating system, and so on

• Severity—info, warning, minor, major, critical

• Direction—traffic direction (client-to-server, server-to-client, or any)

• Attack Type—signature (threat matched against signature), anomaly (threat due to protocol deviation)

• Service Type—predefined TCP and UDP services

• False Positives—criteria if frequently, occasionally, rarely, or unknown for false positives

• Recommended—whether the attack is recommended or not

• Last Modified—date of modification for attack signature (field is deprecated and no longer used)

As new attack objects are added to the attack database by the Juniper Networks Security team, they are automatically

added to existing dynamic groups matching the defining criteria for that group.

A use case for custom attack groups would be an Internet service provider (ISP) or university administrator who wants

to allow peer-to-peer traffic, but protect against peer-to-peer vulnerabilities. This can be achieved by creating a

custom attack group containing filters for Product type Peer-To-Peer and severities Minor, Major, and Critical. Blocking

only this grouping would allow peer-to-peer traffic, but block exploits and attacks due to peer-to-peer. By making this

a dynamic custom attack group, future attack objects matching these criteria would automatically be added to the

dynamic group.



Figure 11: Defining a custom dynamic group

Terminal Rules

Normal IDP Series rule processing occurs in a linear sequence starting with the first rule and processing until the last

rule. For each rule, traffic is matched against source/destination addresses and against the attack signature before

taking the defined action. Sometimes an administrator may want to match traffic against a set of criteria, take an

action on the matched packets, and not subject the traffic to inspection against subsequent IDP Series rules. This

can be useful to avoid multiple logs/alerts if traffic is known to match multiple signatures. This can also be useful to

troubleshoot or prevent false positives.

The “Terminate Match” checkbox creates such a terminal rule. In a terminal rule, traffic matching the source,

destination, and service is not subjected to inspection in subsequent rules. Caution must be used when using terminal

rules, as traffic only has to match the source, destination, and service to be considered terminal. The rule is terminal

even if the traffic does not match the attack. To optimize performance, terminal rules should be placed near the top of

the rulebase.

Terminal Rule Example:

(S)= Source, (D) = Destination, (Se)=Service, (A)= Attack

• Normal rule: Match (S), (D), (Se), (A) -> Take defined action, move to next rule

• Normal rule: Match (S), (D), (Se), but does not match (A) -> No action, move to next rule

• Terminal rule: Match (S), (D), (Se), (A) -> Take defined action, terminate rule processing

• Terminal rule: Match (S), (D), (Se), but does not match (A) -> No action, terminate rule processing



Exempt Rules

The IDP Series rulebase can have attack objects that will match against traffic and produce false positives or irrelevant log

records. Alternatively, there may be specific source or destination addresses that should be excluded from attack detection.

The Exempt rulebase can be used to exclude specific source/destination pairs against specific attack objects using the

following flow of rule processing:

1. Traffic first matched a rule in the IDP Series rulebase.

2. Traffic is then matched against the Exempt rulebase.

3. If not matched in the Exempt rulebase, the specified action/log is carried out.

4. If traffic matches in both the IDP Series rulebase and a rule in the Exempt rulebase, then traffic passes without action/log.

Example:

The following IDP Series rule blocks all chat traffic from the finance network:

The exempt rule would allow MSN traffic to be permitted.

Figure 12: Exempt rules

False Positives

Unintended traffic drops or alerts are considered to be false positives. Tuning the IDP Series security policy to minimize

false positives is best achieved by crafting highly specific rules customized for addresses and specific attack objects.

Depending on the IDP Series’ placement and custom network patterns, some false positives will still be encountered.

The following approaches for handling false positives are recommended:

• Understand the attack object and its application.

• Review the attack objects description within the Object Manager to better understand how to apply the attack

object.

• Exempt the attack object from matching with an exempt rule.

- This will bypass the attack object. Create the exempt rule to be as specific as possible by using a specific source/

destination address pair and the attack object in the exempt rule.

• Collect a packet capture and notify the Juniper Security team.

- Configure a temporary 10 pre-packet and 20 post-packet post-capture under the notification column. If the

capture is TCP- based traffic, it should contain the TCP three-way handshake. Send the packet capture and

suspect attack object to [email protected] for analysis.

Example: Several MS-RPC evasion attack objects will match on SMB print job traffic generating a false positive.

As MS-RPC evasion attacks will appear on traffic originating on the WAN, apply MS-RPC evasion attack objects only

on traffic from the Internet on WAN links. MS-RPC evasion attack objects should not be applied on rules inspecting

internal LAN traffic.



Performance Considerations

With the impressive number of features and options available to inspect traffic with IDP Series appliances, it is

important to optimize the policy and IDP Series tools utilized to maximize IPS effectiveness. As with any network

solution, the IDP Series model deployed has to be appropriate to the traffic levels of the inspected network. The peak

traffic levels should not regularly exceed the IDP Series rated capacity. This section reviews certain aspects of the IDP

Series options that are potentially taxing to performance and discusses approaches to minimize impact.

Optimize Traffic Filtering

One simple approach to improving and optimizing IDP Series policy performance is to direct traffic to relevant attack

objects. By filtering traffic in the rulebase according to source and destination addresses, packets are directed

toward specific attack signatures thereby increasing performance and also reducing false positives. An example rule

implementing this would be to filter traffic inspection for database exploits to traffic destined to an SQL server address

object only. All other traffic would bypass this rule.

Client-to-Server/Server-to-Client

Many IDP Series attack objects are defined to implement detection in a direction of flow when attempting to match

traffic—any, client-to-server, server-to-client. Server-to-client attack objects are resource intensive. The direction of

an attack object can be viewed by opening the attack object and editing it and viewing the detection tab. To improve

IDP Series policy performance, implement server-to-client signatures with specific source/destination address objects

rather than “any” to direct relevant traffic to the signature.

Notifications

Notifications can impact IDP Series performance depending on the rate and type. Performance will vary based on the specific

IDP Series model and traffic levels, but log notifications should be minimized not only to improve performance, but to reduce

the noise in log reports to ease notice of relevant attacks. While increased notification levels are a must while diagnosing

attacks, under normal IDP Series usage, no notifications are recommended for minor, warning, and info severity attacks.

Additionally, packet logging is very resource intensive as packet captures are logged to disk. Pre-packet captures

require IDP Series appliances to capture ahead of attacks and should only be used when diagnosing false positives.

Application Identification

A new feature introduced in standalone IDP Series 4.1, Application Identification (AI) can identify applications running

on dynamic ports by matching patterns in the data stream.

AI allows detection of applications running on non-standard or unknown ports. Previously, signature decoders had

to have port specified with static protocol mapping. Many peer-to-peer (Bittorent, Kaaza) and chat (Skype, Yahoo

Messenger, and so on) applications are designed to use dynamic ports. AI determines application by traffic patterns

and then applies the appropriate signatures independent of the ports being used. It will match a signature based on

the first client-to-server or first server-to-client packet.

Application Identification improves application detection and reduces false positives by not having to rely on static

port definitions in the signature.

It is enabled by default in IDP Series 4.1. The setting to disable is in the Sensor Settings > Load Time Parameters

section within NSM. If AI is disabled, older signatures based on ports and context decoders are used.

Fine-tuning and Monitoring the Network

A key component to fine-tune the IDP Series deployment is to understand the sources and destinations of typical

network traffic patterns on your network and be able to identify unique network events. IDP Series appliances have

several tools that are valuable to analyze and monitor the network . These include Enterprise Security Profiler, SCTOP

Command-Line Utility, and Security Explorer.



Security Profiler

The Profiler is best utilized in the initial deployment to learn about your network and its resources. The Profiler collects

a database of each unique event that occurs on your network, allowing you to identify the following:

• Hosts and servers

• Traffic ports and protocols used

• Layer 3, L4, and L7 data identifying applications, host operating systems, users, and services

Profiler is initially used to create a network baseline where it identifies each host, server, and software application that

regularly appears on the network. As part of establishing a baseline, Profiler builds a database of operating systems,

applications, versions, and other parameters to characterize normal traffic. After a network baseline is established,

Profiler should be configured to alert on network deviations. Profiler can send an alert when a new host or new

application appears on the network.

As an example, Profiler can be instrumental in identifying a new infection of a host with a network-scanning worm.

Profiler quickly identifies the network scan as a non-standard activity that can fire off an alert. Review of the Profiler

database can identify the specific IP and media access control (MAC) address of the infected host, allowing a quick

response to the malware.

Detailed notes on the configuration and application of the Profiler are in the NSM Administrator’s Guide.

Security Explorer

The Security Explorer is a graphical tool allowing the administrator to correlate network traffic based on data collected

in the Profiler, Log Viewer, and Report Manager. It displays several panes that graphically depict relations between

objects based on Peer IP, inbound/outbound services, and client/server profiles as well as attacks. The report tab can

be used to view top attacks, alarms, logs, destination IPs, attacks over time, and attacks by severity.

This correlation allows the administrator to learn traffic patterns within the network that are not immediately evident

and apply policy rules tailored to the context seen.

SCTOP Command-Line Utility

Monitoring IDP Series performance and utilization is easily done using the built-in command-line IDP Series utility

SCTOP. This utility is accessed via SSH in the IPS sensor (standalone IDP Series products only) as admin, and su to

root user. SCTOP is launched via the sctop command-line interface (CLI) command.

Snapshot of SCTOP options:

sctop help ‘h’ - Display this help ‘a’ - ARP/MAC table ‘i’ - IP flows ‘c’ - ICMP flows ‘u’ - UDP flows ‘t’ - TCP flows ‘r’ - RPC table ‘x’ - RPC XID table ‘s’ - Subscriber’s status ‘m’ - Memory statistics ‘l’ - Q-Module statistics‘e’ - Rulebase statistics‘g’ - Aggregate statistics‘k’ - Attack statistics‘p’ - Spanning tree protocol‘b’ - IP Action table‘z’ - Packet distribution‘d’ - Strip Chart‘f’ - Fragment chain‘w’ - HA status‘y’ - IDS cache statistics‘q’ - Quit the program

kernel v4.1.104571-idp41‘v’ - reverse sort order‘0’ - disable sorting‘1’ - sort by bytes/session‘2’ - sort by packets/session‘3’ - sort by expiration‘4’ - sort by service‘5’ - sort by dst port‘6’ - sort by src ip‘7’ - sort by dst ip‘8’ - sort by vlan



The following SCTOP displays are the most useful for IDP Series monitoring:

• s—Subscriber status: Very useful to monitor IDP Series network throughput, policy, traffic peaks, uptime, version

• k—Displays attack statistics in order of frequency of hits

• i, t, u—Displays IP, TCP, and UDP flows, respectively

• g—Displays Aggregate Statistics based on sessions per IP address

• z—Packet size distribution

Summary

By implementing the approach presented in this solution design guide, a security administrator can simplify the design

and maintenance of the IDP Series policy. Using the newly developed recommend policy and recommended action

features allows the administrator to leverage the knowledge and resources of the Juniper Networks Security team to

better utilize IDP Series appliances. As with any network tool, IDP Series Intrusion Detection and Protection Appliances

have to ultimately be customized to the specific network requirements. The detailed review of IDP Series features and

implementation recommendations presented in this guide will enable the administrator to decrease false positives and

improve overall performance while minimizing ongoing maintenance of IDP Series appliances.

Appendix A: Resources for Ongoing Maintenance

Daily Signature Updates

The Juniper Networks Security team constantly monitors for the latest vulnerabilities and threats and creates

signatures on a daily basis against these threats. Customers can subscribe to the signature bulletins from the support

Web page or the following link can be used directly:

www.juniper.net/alerts/subscribe.jsp?actionBtn=Modify

NSM can be configured to automatically download the attack database on a regular basis. This procedure is detailed in

the NSM Administrator’s Guide in the “Managing Devices > Managing the Attack Database” section.

Detector Updates

Upgrading the attack database in NSM is the one step for IDP Series maintenance. In addition, the Juniper Networks

Security team regularly updates the IDP Series Detector Engine. Updating the detector is done through NSM.

http://kb.juniper.net/KB9773

Detector Engine updates can contain fixes for the protocol decoders, so upgrading to the latest Detector Engine can

solve packet loss conditions and device crashes.

Improvements that go into a new detector include:

• False positive fixes

• Decoder fixes (fix false positives and false negatives)

• New contexts (improve accuracy and performance)

• New protocol decoders (improve performance and accuracy)

• Stability and memory improvements

8010068-001-EN Dec 2009

Copyright 2009 Juniper Networks, Inc. All rights reserved. Juniper Networks, the Juniper Networks logo, Junos, NetScreen, and ScreenOS are registered trademarks of Juniper Networks, Inc. in the United States and other countries. All other trademarks, service marks, registered marks, or registered service marks are the property of their respective owners. Juniper Networks assumes no responsibility for any inaccuracies in this document. Juniper Networks reserves the right to change, modify, transfer, or otherwise revise this publication without notice.

EMEA Headquarters

Juniper Networks Ireland

Airside Business Park

Swords, County Dublin, Ireland

Phone: 35.31.8903.600

EMEA Sales: 00800.4586.4737

Fax: 35.31.8903.601

APAC Headquarters

Juniper Networks (Hong Kong)

26/F, Cityplaza One

1111 King’s Road

Taikoo Shing, Hong Kong

Phone: 852.2332.3636

Fax: 852.2574.7803

Corporate and Sales Headquarters

Juniper Networks, Inc.

1194 North Mathilda Avenue

Sunnyvale, CA 94089 USA

Phone: 888.JUNIPER (888.586.4737)

or 408.745.2000

Fax: 408.745.2100

www.juniper.net

To purchase Juniper Networks solutions,

please contact your Juniper Networks

representative at 1-866-298-6428 or

authorized reseller.

Printed on recycled paper



Known Issues Update

Juniper Networks Technical Assistance Center (JTAC) publishes this update once a month to update customers of

known issues and workarounds on the latest IDP Series releases. Customers can subscribe to Known issues Update

(KIU) from this page:

www.juniper.net/alerts/subscribe.jsp?actionBtn=Modify

Attack Object Information

Each signature is described at this link:

https://services.netscreen.com/restricted/sigupdates/nsm-updates/HTML/index.html

Customers can also refer to the Juniper Security RSS feed, as all updates are listed here:

https://services.netscreen.com/restricted/sigupdates/nsm-updates/updates.xml

The latest CVE to IDP signature mapping file is on the TAC Software Download Pages:

https://services.netscreen.com/restricted/sigupdates/nsm-updates/CVE-BID-mapping.csv

About Juniper Networks

Juniper Networks, Inc. is the leader in high-performance networking. Juniper offers a high-performance network

infrastructure that creates a responsive and trusted environment for accelerating the deployment of services and

applications over a single network. This fuels high-performance businesses. Additional information can be found at

www.juniper.net.

http://www.juniper.net/alerts/subscribe.jsp?actionBtn=Modify

https://services.netscreen.com/restricted/sigupdates/nsm-updates/HTML/index.html

https://services.netscreen.com/restricted/sigupdates/nsm-updates/updates.xml

https://services.netscreen.com/restricted/sigupdates/nsm-updates/CVE-BID-mapping.csv

http://www.juniper.net

idp series policy design and optimization - juniper …€¦ · juniper networks idp series...

Documents