Mitigating Worm AttacksEVENING SEMINAR

Deniz KayaNew Horizons Bulgaria

Agenda

• Introduction• Experience• Incident Response• Worm Mitigation Reaction Methodology• Tools and Techniques• Applying the tools to Enterprise Environment• Appendix

Introduction

• Internet worms have had a severe impact on many enterprise customers. Recently developed tools and architectural techniques can be employed to assist with the mitigation of worm activity in an enterprise environment.

• Here we will speak about:– A conceptual overview of worm mitigation techniques– Details for deployment of these techniques into an overall solution

for enterprise customers

• This seminar was prepared from a solution standpoint. It is primarily designed to provide a tool kit for dealing with the issue of Internet worms within an enterprise environment. Although this is the primary motivation, the overall solution has application well beyond this primary purpose and additionally provides capability for detecting and responding to other security incidents.

Experience

• The techniques described here were originally developed for large Internet service providers (ISPs) and have been adapted for use in enterprise environments. They are well-understood and mature technologies, now applied in a new way to solve a new problem.

• Cisco uses the same techniques on its own network to defend against a range of malicious activity, including worms and other security incidents.

Incident Response

• An organization’s internal operational processes are a critical aspect of dealing with any security incident.

The overall goal of an incident response process is to maintain business operations.

Incident ResponsePreparation• Although preparation is not part of the formal incident response process, here

are some techniques that must be in place prior to the occurrence of a security incident. Having response procedures in place facilitates efficient response during an actual incident.

– The Cisco Network Consulting Engineers suggest the following preparatory steps:

• Develop a clear understanding of the organization’s primary business and IT resources.

• Arrange for 24x7 access to someone who can authorize business decisions during a security incident.

• Establish open lines of communication. Operations groups need to know the key contacts within the organization.

• Collect links to Internet sites that provide up-to-date and reliable details of security threats and Internet worm activity, such as www.dshield.org, www.securityfocus.com, and bugtraq.

• Maintain updated contact details for your ISP or ISPs.

Incident Response:Triage: Initial Analysis and Response• The first phase of incident response is to verify that the event is an actual

security incident, such as an attack or worm event. In some cases, an incident could be the result of scheduled maintenance activities.

• After the event is confirmed, take quick action to limit the damage. Doing so might entail steps such as turning off a device or removing a device from the network. However, any actions taken need to be in line with maintaining business continuity.

• During the process, communicate with other relevant parties within the organization. For example, stay in touch with relevant management and legal contacts.

Incident Response:Analysis

• The second phase is the analysis phase.

• Next, determine the scope of the incident-the number of devices, data, and other resources affected.

• In some cases, it might be necessary to perform a traceback to the origin of the attack; this activity might involve working through your ISP.

• Measure the impact.

• The results of this analysis will help determine the most appropriate reaction techniques for the specific incident.

Incident Response:Reaction

• The reaction phase involves some action to counter the attack. Each situation will dictate the action to be taken, such as widely deploying access control lists (ACLs) in a worm event; restoring a device to normal operation by reloading the OS from the original media and restoring data from backups in a server compromise; or changing any static passwords because they might have been compromised-and an entirely reasonable response in some situations might be to do nothing.

• Generally, the highest priority is to regain full business operations. In many cases it is often less important to spend time finding the perpetrator of the attack.

Incident Response:Post-Mortem

• A post-mortem involves a full, in-depth analysis of the event and the response to the event. The goal is to determine what can be done to build resistance and prevent this type of attack from happening again-essentially, learning from the experience.

• The post-mortem is a step that is often ignored. It is critical that it is not forgotten.

Worm Mitigation Reaction Methodology

• Following procedures should be followed when responding to a worm incident– Containment– Inoculation– Quarantine– Treatment– Planning

Worm Mitigation Reaction Methodology:Containment

• The first stage of the reaction process is to contain the spread of the worm inside the network. Compartmentalization, a core principle of the SAFE Blueprint from Cisco, is key because it allows isolation of parts of the network that are not yet infected.

Worm Mitigation Reaction Methodology:Inoculation

• The inoculation phase involves patching all systems. If the appropriate signature files or plug-ins are available for tools such as OpenVAS, it is worthwhile to start scanning the network for vulnerable systems. This activity might allow operations staff to find vulnerable systems before they become infected.

• During a worm crisis, there are three types of systems in your network:– Patched systems– Unpatched systems– Infected systems

• Inoculating uninfected systems is imperative and usually happens in parallel with the quarantine and treatment phases.

Worm Mitigation Reaction Methodology:Quarantine

• The quarantine phase involves finding each infected machine and disconnecting, removing, or blocking them from the network to prevent them from infecting other unpatched machines on the network. To achieve this goal, the infected systems need to be isolated and quarantined.

• Later in this seminar we will outline tools such as remote-triggered black hole routing. This technique allows the rapid isolation of infected machines, limiting their capability to spread the infection.

Worm Mitigation Reaction Methodology:Treatment

• The treatment phase involves the cleaning and the patching of each infected system. Some worms might require complete reinstallations of the core system to ensure that the machine is clean.

Worm Mitigation Reaction Methodology:Planning

• All of this activity requires planning prior to a worm event. When these events occur, reaction time is critical, and these processes need to be in place. It is strongly recommended that every organization plan the reaction methodology ahead of the next crisis.

Tools and Techniques

• It is important to view the following techniques as a tool kit. There is currently no simple guaranteed solution for dealing with these types of security incidents.

• The main tools we will discuss here are:• Features

– ACLs– NetFlow and NetFlow export– Unicast Reverse Path Forwarding (uRPF)– Routing protocols such as remote-triggered black hole filtering, also known as

remote-triggered black hole routing• Products

– Cisco routers and switches– NetFlow collectors– Arbor Networks Peakflow X and Peakflow DoS

• There are many other products and features that can be used as security tools. Here we are only speak a subset of these tools to help you orientate.

Tools and Techniques:ACLs (Cont.)• ACLs as Security Tools

ACLs serve a dual purpose as security tools. They provide:

– A mechanism to permit or deny traffic– A mechanism to detect certain traffic types

The use of ACLs to permit or deny traffic is a well-understood and well-documented security feature. In terms of worm mitigation, ACLs are likely to play a key role in preventing the spread of a worm by blocking its attack vector, usually a TCP or UDP port.

Tools and Techniques:ACLs (Cont.)• Using ACLs as a Detection Tool

– The most common technique when using ACLs as a detection tool is to configure the router as a pseudo packet sniffer. To do so, use an ACL with a series of permit statements to provide a view of the traffic flow. The counters in the ACL entries can then be used to find which protocol types are potential culprits.

Tools and Techniques:ACLs • VLAN ACLs

– VLAN access control lists (VACLs) operate somewhat like router-based ACLs. They are a means to apply access control to packets bridged within a VLAN or routed between VLANs. In terms of worm mitigation, VACLs allow access control to be applied directly to the access port.

– VACLs use the same Access Control Entry (ACE) format used by router-based ACLs. The permit and deny statements based on Layer 2-4 header information are used to determine what traffic to permit and to deny. VACLs have no sense of direction, unlike router-based ACLs, which are applied on either an inbound or outbound basis. VACLs apply to traffic at both ingress and egress.

Tools and Techniques:NetFlow• NetFlow is used as the foundational technology for obtaining traffic flow

information across a network. A flow is defined by seven unique keys: source IP address, destination IP address, source port, destination port, Layer 3 protocol type, ToS byte, and input logical interface (ifIndex).

• By observing traffic flows across the network, it is possible to see events that might be malicious. Some events might cause high traffic volumes, such as a denial of service (DoS) attack; others might be more subtle. In any case, observation of the flow information can detect these events

Tools and Techniques: NetFlow (Cont.)• NetFlow has the capability of performing a flow export function. In this case, all

expired flow information is sent to a collector. Collectors could be a number of devices, including a Cisco NetFlow Collector, CFLOWD tools, OSU flow-tools (CFLOWD Successor), or the Arbor Networks collector.

Tools and Techniques: NetFlow• The current NetFlow information is also available via the command-line

interface (CLI) of the router. The sample output shows two clients infected with the Blaster worm that are scanning for other systems to infect. Note: 0x87 equals port 135 (illustrated in pink below).

Tools and Techniques: NetFlow Deployment (Cont.)• NetFlow monitors an interface’s ingress traffic only. Therefore, to obtain a full

picture of bidirectional flow information, NetFlow must be deployed such that all ingress and egress flows are capturedv

Tools and Techniques: NetFlow Deployment (Cont.)• Performance Impact

– NetFlow will have some performance impact. The largest dependency from a performance perspective is the number of flows. The performance impact needs to be assessed on a case-by-case basis. In worst-case scenarios, router upgrades might be required.

• Collection Tools– There are many options for collecting exported NetFlow information. A

commercial option is the Cisco CNS NetFlow Collection Engine. This can be deployed on a number of platforms, including Solaris, HP UX, and Linux.

– Freeware tools are also available. The OSU flow-tools from Oregon State University are essentially the successor of CFLOWD and are available at:

http://www.splintered.net/sw/flow-tools/

• Exporting and Analyzing Flow Information for Anomalies– Arbor Networks Peakflow provides further details of how the Arbor

Peakflow products integrate into the overall solution. Additional NetFlow Information

Tools and Techniques:Arbor Networks Peakflow (Cont.)• Peakflow Overview

• The detection and recognition of an attack or a security event is a critical component of any security solution.

• Although IDSs provide detection capability, most of them are still signature-based, and therefore of limited benefit in these situations. Cisco itself has used the Arbor Peakflow DoS anomaly detection system to successfully detect and mitigate several worms.

Tools and Techniques:Arbor Networks Peakflow (Cont.)• Arbor offers two solutions to this problem.

– Peakflow DoS• The primary application of Peakflow DoS is the detection of external

threats and events, making this product widely deployed by ISPs. For enterprises, using Peakflow DoS to detect the presence of an external security event (an event outside the firewall) is key to being in a position to quickly secure the network "internally" from the threat.

• In the context of this solution, Peakflow DoS would be used as a tool used to monitor traffic outside an organization’s firewall.

– Peakflow X• The primary application of Peakflow X is the detection of internal

threats and events. Peakflow X provides an internal anomaly detection solution through relational modeling of the enterprise’s internal network.

• In the context of this solution, Peakflow X provides a detailed visualization of the application-level conversations inside an enterprise network.

Tools and Techniques:Arbor Networks Peakflow (Cont.)• Placement of the Arbor Collectors

– Both Arbor Peakflow X and Peakflow DoS use a collector and controller architecture. The Arbor collector receives the flow records exported from the routers. Multiple routers can export flow information to a single collector. A controller provides a Web interface, sits in the hierarchy above the collectors, and generally consolidates the information from the controllers.

Tools and Techniques:Sinkholes (Cont.)• A sinkhole is a multifaceted security tool-essentially, a portion of the network

that is designed to accept and analyze attack traffic.• In the first sinkhole application, a publicly accessible Web server is the target of

either a DoS or DDoS attack. Below we see how server WWW1 is unavailable due to the attack. Additionally, the extremely high traffic volume has saturated links and routers, making server WWW2 unavailable as well.

Tools and Techniques:Sinkholes (Cont.)• Here we can see how a sinkhole can be used to pull attack traffic destined for

WWW1 away from the target.

• A sinkhole is also a useful tool for analyzing an attack. The sinkhole router can be used to forward the attack traffic to a back-end switch where a network analyzer, such as a sniffer or Ethereal, can be used to look at the details of the attack.

Tools and Techniques:Sinkholes – Monitoring the Worm Propagation

– Here we can see how a sinkhole can be deployed to monitor for worm propagation internally within an enterprise.

Although this example specifically illustrates the application of a sinkhole for detecting worm propagation, monitoring the bogon and dark IP address space can also detect other usually malicious activity.

Tools and Techniques:Sinkholes – Backscatter Traffic

• Packets with unreachable destinations, including the router null0 interface, will have an Internet Control Message Protocol (ICMP) unreachable message sent back to the source address. This "unreachable noise" is known as backscatter. A sinkhole is likely to draw in a substantial amount of backscatter traffic. This is particularly true for Internet-based sinkholes.

Backscatter traffic on the Internet is often the result of large-scale DoS or DDoS attacks in which spoofed source addresses have been used.

Tools and Techniques:Sinkholes – Deployment Option 1

• In this scenario, the target router on the right might be a low-cost device, possibly a Cisco 2600 or 3600 series router. Its primary purpose is to gather and export NetFlow information.

• Routing announcements for the bogon and dark IP address space can be made from either the target router or the sinkhole gateway.

Tools and Techniques:Sinkholes – Deployment Option 2

• The second design option uses some form of dedicated high-speed router. • A second Ethernet interface should be available on this router for both NetFlow

export and dedicated Simple Network Management Protocol (SNMP) polling.

As in the first option, bogon and dark IP address space is announced from the sinkhole router, preferably via the redistribution of static routes. The static routes will use a bogus next hop and a static ARP entry to push traffic onto the switched network.

ip route 96.0.0.0 63.255.255.255 192.0.2.200 ip arp 192.0.2.200 00.00.0c.12.34.56 arpa

Tools and Techniques:Black Hole Routing• A black hole routing scheme is based on the concept of forwarding traffic to

null0. The technique achieves a similar result to an ACL based on destination address. However, because the technique occurs directly in the forwarding (or Cisco Express Forwarding) path, it achieves a dropping function with no performance impact.

Tools and Techniques:Remote-Triggered Black Hole Routing

• Although black hole routing is an effective technique for dropping traffic at line rates, we need to add remote trigger capability. This is achieved with two steps.

• The first step is to configure an unused route to null0. This needs to be configured on all routers that will act as remote-trigger black hole routers. For example: ip route 192.0.2.0 255.255.255.0 Null0192.0.2.0 /24 is an unused address block called the Test-Net. As such, it is not publicly allocated and is often used for this application.

• In the second step, Border Gateway Protocol (BGP) is used to propagate information about a prefix we want to black hole.

Tools and Techniques:Remote-Triggered Black Hole Routing

• After the trigger router is in place, a configuration like the one below is typically used to announce the prefixes that should be black holed.

• router bgp 999• ...• redistribute static route-map STATIC-TO-BGP• ...• !• route-map STATIC-TO-BGP permit 10• match tag 66• set ip next-hop 192.0.2.1• set local-preference 50• set origin igp• !• Route-map STATIC-TO-BGP permit 20• !• ...• ip route 171.xxx.xxx.1 255.255.255.255 Null0 Tag 66 • !

Tools and Techniques:Dropping on Source Address

• One of the criteria for remote-triggered black hole routing to be effective as a security tool is the ability to drop traffic based on both destination address and source addresses.

• A second scenario requiring a mitigation technique is one in which spoofed source addresses are used. With recent worms, such as SQL Slammer and Blaster, the host’s real IP address is used to propagate the worm. This is not to say that other worms might not use spoofed addresses. As such, the scenario needs to be accommodated. There is no reason that any host should ever send out a packet with an address other than what was assigned to it. Any packets being sent out with illegitimate source addresses should be dropped at the first router hop.

Tools and Techniques:Dropping on Source Address

• Unicast RPF in Strict Mode :If a packet is received on an interface, a route to that packet’s source address must be available back through the same interface on which the packet was received. If this route does not exist, the packet fails the RPF check and is dropped.

interface FastEthernet2/0 ip address 192.xxx.xxx.50 255.255.255.0 ip verify unicast reverse-path

Tools and Techniques:Dropping on Source Address

• Unicast RPF in Loose Check ModeIn the case of loose check, the only requirement is that the source address must appear in the router’s Cisco Express Forwarding table. If the route does not exist or it has a destination of null0, the packet is dropped.

interface FastEthernet2/0 ip address 192.xxx.xxx.50 255.255.255.0 ip verify unicast source reachable-via any

Tools and Techniques:Dropping on Source Address

• Selective Remote Traffic DroppingThe previous sections on NetFlow and sinkholes provided a set of techniques for identifying infected machines and listed a variety of abnormal behaviors that might represent a security incident. When an infected machine or security event is identified, the operations staff has the option of black holing the device.

ip route xxx.xx.xxx.242 255.255.255.255 Null0 Tag 66 ip route xxx.xx.xxx.204 255.255.255.255 Null0 Tag 66

Tools and Techniques:Private VLANs• Private VLANs are a technique for providing Layer 2 isolation of hosts within a

VLAN. This technique can improve the security posture of a network by isolating servers that do not need to communicate with each other. From a security standpoint, if one server were to become infected with a worm, its inability to communicate with other servers would prevent the spread. In this case, each server would be attached to an isolated port.

Tools and Techniques:Other Quarantine Techniques• Port control using scripting• Policy-based routing• Web Cache Communication Protocol• MAC addresses• 802.1x• Remote access

Appendix

• Aggregated Bogon Listhttp://www.cymru.com/Bogons/index.html

• Freeware Tools– http://www.net-snmp.org/ – http://www.cpan.org/ – http://oss.oetiker.ch/mrtg/ – http://oss.oetiker.ch/rrdtool/ – http://www.splintered.net/sw/flow-tools/ – http://net.doit.wisc.edu/~plonka/FlowScan/

Q and A