service provider responses to denial of service attacks · copyright © 2003, cisco systems, inc....
TRANSCRIPT
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.8172_05_2003_c1
2© 2003, Cisco Systems, Inc. All rights reserved.SEC-20088127_05_2003_c1
Service Provider Responses to Denial of Service Attacks
Session SEC-2008
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.8172_05_2003_c1
333© 2003, Cisco Systems, Inc. All rights reserved.SEC-20088127_05_2003_c1
News—January 22, 2002Cloud-Nine Officially Closes ISP!
By:mark.j @ 10:44:AM—Comments (35)—SendNews [HERE]/PrintNews [HERE]
• Today looks set to be a sad and frustrating one for anybody who was ever a customer of the once popular unmetered dialup and broadband ISP Cloud-Nine
• At precisely 10:16 am a few minutes ago Emeric Miszti (CEO) and John Parr (Operations Director) of the C9 ISP posted what’s likely to be their final announcement on our forums; C9 is now the latest ISP to close, although it's the first we've ever seen to go from a hack attack!:
Cloud Nine regret to announce that at 7:45 this morning the deci sion was taken to shut down our Internet connections with immediate effect
We tried overnight to bring our web servers back online but were seeing denial of service attacks against all our key servers, including email and DNS; these were of an extremely widespread naturehttp://www.ispreview.co.uk/cgi -bin/ispnews/printnews.cgi?newsid1011696274,91619
444© 2003, Cisco Systems, Inc. All rights reserved.SEC-20088127_05_2003_c1
IntrusionUnauthorized User…
DoSAuthorized User…
…Gets No Access…Gets Access
What Is Denial of Service?
Location of Attacks: Layer 1–7
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.8172_05_2003_c1
555© 2003, Cisco Systems, Inc. All rights reserved.SEC-20088127_05_2003_c1
DoS: The Problem
• We understand intrusions: Do things right, and you’re okay
• What about DoS? Do things right, and still get drowned…
• Here: Focus on ISP issues with DoS
666© 2003, Cisco Systems, Inc. All rights reserved.SEC-20088127_05_2003_c1
3. Flooding
mbehring
DoS: The Procedure
ISP CPE Target
“Zombies”or “Bots”
Hacker
1. Cracking2. Signalling
InnocentUser PCs
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.8172_05_2003_c1
777© 2003, Cisco Systems, Inc. All rights reserved.SEC-20088127_05_2003_c1
DoS: The Mechanisms Used
1. Cracking:
Manually, through viruses, worms (sql slammer)…; often exploiting host vulnerabilities
2. Signalling:
IRC, specific ICMPs, through “relays”…
3. Flooding:
TCP SYN flood, UDP, ICMP, other IP protocols…
Attacking a Line: Big Packets (Bandwidth!)
Attacking a Host/Router: Small Packets (pps!)
888© 2003, Cisco Systems, Inc. All rights reserved.SEC-20088127_05_2003_c1
Agenda
• Detecting and Classifying DoS Attacks
• Tracing DoS Attacks
• Containing DoS Attacks
Disclaimer: DoS Is a Research Topic!
Please Contribute Your Experience!
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.8172_05_2003_c1
9© 2003, Cisco Systems, Inc. All rights reserved.SEC-20088127_05_2003_c1
Detecting and Classifying DoS Attacks
101010© 2003, Cisco Systems, Inc. All rights reserved.SEC-20088127_05_2003_c1
Ways to Detect and Classify DoS Attacks
• Customer call
• SNMP: Line/CPU overload, drops
• NetFlow: Counting flows
• ACLs with logging
• Backscatter
• Sniffers
• Arbor’s Peakflow DoS (partner product)
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.8172_05_2003_c1
111111© 2003, Cisco Systems, Inc. All rights reserved.SEC-20088127_05_2003_c1
Detecting DoS through CPU Load
• A: Total CPU load
• B: CPU at interrupt level (note: B <= A)
• A-B: Process switched traffic, CPU processes
A: CPU Total Utilization B: CPU at Interrupt Level
router>sh proc cpu
CPU utilization for five seconds: 99%/97%; one minute: 78%; five minutes: 23%
(See: http://www.cisco.com/warp/public/63/highcpu.html)
121212© 2003, Cisco Systems, Inc. All rights reserved.SEC-20088127_05_2003_c1
router>sh proc cpu
CPU utilization for five seconds: 99%/97%; one minute: 78%; five minutes: 23%
Abnormal CPU Load
• If A B: “Too much traffic to forward”Interrupts: Packet switching (fast switching)
• If A >> B: “Too much central processing”Packets to/from the router (e.g. SNMP, ICMPs, vty and console, IPsec (w/o h/w), routing…) Process switched packets or switching problemIf attack, targeted at the router itself!
A: CPU Total Utilization B: CPU at Interrupt Level
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.8172_05_2003_c1
131313© 2003, Cisco Systems, Inc. All rights reserved.SEC-20088127_05_2003_c1
Count Flowswith Sampling 1/Y
During Z Sec
Alarm!
On Border Routers, Every X Min:
Detecting DoS Attacks with NetFlow
• Basis: Have NetFlow running on the network
DANTE Uses: X=15 Min, Y=200, Z=10 Sec, N=10
Values Are EmpiricalIf
# of Flows> N
End
Y
N
See Arbor Networks Section Later in this Presentation
141414© 2003, Cisco Systems, Inc. All rights reserved.SEC-20088127_05_2003_c1
Real data deleted in this presentation
Real data deleted in this presentation
Potential DoS Attack (33 Flows) on Router1 Estimated: 660 Pkt/s 0.2112 Mbps ASxxx Is: …ASddd Is: …
src_ip dst_ip in out src dest pkts bytes prot src_as dst_asint int port port
192.xx.xxx.69 194.yyy.yyy.2 29 49 1308 77 1 40 6 xxx ddd192.xx.xxx.222 194.yyy.yyy.2 29 49 1774 1243 1 40 6 xxx ddd192.xx.xxx.108 194.yyy.yyy.2 29 49 1869 1076 1 40 6 xxx ddd192.xx.xxx.159 194.yyy.yyy.2 29 49 1050 903 1 40 6 xxx ddd192.xx.xxx.54 194.yyy.yyy.2 29 49 2018 730 1 40 6 xxx ddd192.xx.xxx.136 194.yyy.yyy.2 29 49 1821 559 1 40 6 xxx ddd192.xx.xxx.216 194.yyy.yyy.2 29 49 1516 383 1 40 6 xxx ddd192.xx.xxx.111 194.yyy.yyy.2 29 49 1894 45 1 40 6 xxx ddd192.xx.xxx.29 194.yyy.yyy.2 29 49 1600 1209 1 40 6 xxx ddd192.xx.xxx.24 194.yyy.yyy.2 29 49 1120 1034 1 40 6 xxx ddd192.xx.xxx.39 194.yyy.yyy.2 29 49 1459 868 1 40 6 xxx ddd192.xx.xxx.249 194.yyy.yyy.2 29 49 1967 692 1 40 6 xxx ddd192.xx.xxx.57 194.yyy.yyy.2 29 49 1044 521 1 40 6 xxx ddd… … … … … … … … … … …
What Does a DoS Attack Look Like?
Real Data Deleted in this Presentation
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.8172_05_2003_c1
151515© 2003, Cisco Systems, Inc. All rights reserved.SEC-20088127_05_2003_c1
• Requires ACLs to be in place (for detection)Extended IP access list 169
permit icmp any any echo (2 matches)
permit icmp any any echo-reply (21374 matches)
permit udp any any eq echo
permit udp any eq echo any
permit tcp any any established (150 matches)
permit tcp any any (15 matches)
permit ip any any (45 matches)
? Watch performance impact? Used on demand, not pro-active? More used for checking than for detection? Some ASIC based LCs do not show counters
Classifying DoS with ACLs
Looks Like Smurf Attack
Found:Found:•• Attack typeAttack type•• InterfaceInterface
161616© 2003, Cisco Systems, Inc. All rights reserved.SEC-20088127_05_2003_c1
Code Red Worm (Aug 2001)
• Infects Microsoft IIS web servers
• Spread: Using real source, random destination
• Attack: accessing a specific server
http get Fill Buffer Unicode EncodedAssembler Code
Infected Host(Real IP!)
217.33.138.14- - [07/Aug/2001:01:17:32 +0100] "GET /default.ida?NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN%u9090%u6858%ucbd3%u7801%u9090%u6858%ucbd3%u7801%u9090%u6858%ucbd3%u7801%u9090%u9090%u8190%u00c3%u0003%u8b00%u531b%u53ff%u0078%u0000%u00=a HTTP/1.0" 400 328
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.8172_05_2003_c1
171717© 2003, Cisco Systems, Inc. All rights reserved.SEC-20088127_05_2003_c1
Code Red: Detection of Infected Hosts
• Spread: Infected host accesses random IP addresses (using real IP address)
• Examine Scatter: Every host likely to receive some “code red” HTTP requests!
• On an ISP’s network: Route big chunks of unused space à sniffer
Sniffer receives lots of code red http connects
Log IP sources: These hosts are infected!
181818© 2003, Cisco Systems, Inc. All rights reserved.SEC-20088127_05_2003_c1
SQL Slammer Worm (Jan 2003)
• Very fast: Maximum spread after 10 min!
• No exploit, just spread (lucky!!)Could have erased disks on all systems!
• Using true source, random destinations
• High pps load on the networks
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.8172_05_2003_c1
191919© 2003, Cisco Systems, Inc. All rights reserved.SEC-20088127_05_2003_c1
Backscatter Analysis
• Sink hole router: Statically announce unused address space (1/8, 2/8, 5/8, …) (see http://www.iana.org/assignments/ipv4-address-space)
• Note: Hackers know this trick: Use also unused space from your own ranges!!!
• Or, use default (if running full routing)
• Victim replies to random destinations
• à Some backscatter goes to sink hole router, where it can be analyzed
202020© 2003, Cisco Systems, Inc. All rights reserved.SEC-20088127_05_2003_c1
Backscatter Analysis
Target
IngressRouters
OtherISPs
Random Sources
Random Sources
Random Sources
Random Sources
Sink Hole Router
Random Destinations
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.8172_05_2003_c1
212121© 2003, Cisco Systems, Inc. All rights reserved.SEC-20088127_05_2003_c1
Advanced Sink Hole Design
Packet Capture/IDS
1 or More Hosts: Spoofing Services: Responds to ARP/IP Services for
Advertised Space
Flow of Mgmt Data
Analysis Segment
Avoid Dynamic Routing
Backbone
Sink Hole Router
222222© 2003, Cisco Systems, Inc. All rights reserved.SEC-20088127_05_2003_c1
Source: http://isc.incidents.org/analysis.html?id=167
Case Study: Slapper Worm (Sep 2002)
int isreal(unsigned long server) {. . .
if (a == 127 || a == 10 || a == 0) return 0;
if (a == 172 && b >= 16 && b <= 31) return 0;
if (a == 192 && b == 168) return 0;return 1;
}
. . .if (!isreal(udpclient.in.sin_addr.s_addr)) break;. . .
IP Address: a.b.0.0
Worm does not use: 127.x.x.x
10.x.x.x0.x.x.x
172.16-31.x.x192.168.x.x
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.8172_05_2003_c1
232323© 2003, Cisco Systems, Inc. All rights reserved.SEC-20088127_05_2003_c1
Re-Directing Traffic from the Victim
Sink Hole Router: Announces Route “target/32”
Target
IngressRouters
OtherISPs • Keeps line to customer clear
• But cuts target host off completely
• Discuss with customer!!!
24© 2003, Cisco Systems, Inc. All rights reserved.SEC-20088127_05_2003_c1
Tracing DoS Attacks
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.8172_05_2003_c1
252525© 2003, Cisco Systems, Inc. All rights reserved.SEC-20088127_05_2003_c1
Tracing DoS Attacks
• If source prefix is not spoofed:à Routing table à Internet Routing Registry (IRR)à Direct site contact
• If source prefix is spoofed: à Trace packet flow through the networkà Find upstream ISPà Upstream needs to continue tracing
262626© 2003, Cisco Systems, Inc. All rights reserved.SEC-20088127_05_2003_c1
madrid% whois -h whois.arin.net 64.103.0.0
OrgName: Cisco Systems, Inc. OrgID: CISCOS-2Address: 170 West Tasman DriveCity: San JoseStateProv: C APostalCode: 95134Country: US
NetRange: 64.100.0.0 - 64.104.255.255 CIDR: 64.100.0.0/14, 64.104.0.0/16 [...]
TechHandle: CAH5-ARINTechName: Huegen, Craig TechPhone: +1-408-526-8104TechEmail: [email protected]
OrgTechHandle: DN5-ORG-ARINOrgTechName: Cisco Systems, Inc. OrgTechPhone: +1-408-527-9223OrgTechEmail: [email protected]
The Internet Routing Registry (IRR): Network Info
• Europe:whois.ripe.net
• Asia-Pac:whois.apnic.net
• USA and rest:whois.arin.net
Contact Information
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.8172_05_2003_c1
272727© 2003, Cisco Systems, Inc. All rights reserved.SEC-20088127_05_2003_c1
madrid% whois -h whois.arin.net as109
OrgName: Cisco Systems, Inc. OrgID: CISCOS-2 Address: 170 West Tasman DriveCity: San JoseStateProv: C APostalCode: 95134Country: US
ASNumber: 109 ASName: CISCOSYSTEMS ASHandle: AS109 […]
TechHandle: MRK4-ARINTechName: Koblas, Michelle TechPhone: +1-408-526-5269TechEmail: [email protected]
OrgTechHandle: DN5-ORG-ARINOrgTechName: Cisco Systems, Inc. OrgTechPhone: +1-408-527-9223OrgTechEmail: [email protected]
The Internet Routing Registry (IRR): AS Info
• Europe:whois.ripe.net
• Asia-Pac:whois.apnic.net
• USA and rest:whois.arin.net
Also, if Domain Known:abuse@domain
282828© 2003, Cisco Systems, Inc. All rights reserved.SEC-20088127_05_2003_c1
IP Source Tracker
• Traditional way of tracking DoS: ACL or NetFlow
Limitation in performance and cross LC support
• Source Tracker: Across LCs, low performance impact
• Availability:GSR E0,1,2,4: Since 12.0(21)S
GSR E3: Planned 12.0(26)S
GSR E4+: Since 12.0(21)S (POS), (23)S (other)
7500: From 12.0(22)S
Other platforms to follow
Line Card
Status as of Apr 2003
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.8172_05_2003_c1
292929© 2003, Cisco Systems, Inc. All rights reserved.SEC-20088127_05_2003_c1
IP Source Tracker
i/f
IN
OUT
i/f
IN
OUT
Switch
GSR
GRP
Packets to the VictimOther Packets
CacheCache CacheCache
Export
Special CEFAdjacency for Victim
Special CEFAdjacencyfor Victim
303030© 2003, Cisco Systems, Inc. All rights reserved.SEC-20088127_05_2003_c1
Ingress Interface
IP Source Tracker: Config
Enable the Feature
Victim
Router# ip source-track <victim>
Router# show ip source-track 10.1.2.1 (also: … summary)Address SrcIF Bytes Pkts Bytes/s Pkts/s10.1.2.1 Pos 1/0 2000 20 200 1
Protocol Total Flows Packets Bytes Packets Active(Sec) Idle(Sec)-------- Flows /Sec /Flow /Pkt /Sec /Flow /FlowICMP 100 1 10 100 10 0 5
See: http://www.cisco.com/en/US/products/sw/iosswrel/ps1829/products_feature_guide09186a00800e9d38.html
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.8172_05_2003_c1
313131© 2003, Cisco Systems, Inc. All rights reserved.SEC-20088127_05_2003_c1
The Flows Come from Serial 1
Tracing Back with NetFlow
Victim
Find the UpstreamRouter on Serial 1
Continue on this Router
router1#sh ip cache flow | include <destination>
Se1 <source> Et0 <destination> 11 0013 0007 159
…. (lots more flows to the same destination)
router1#sh ip cef se1
Prefix Next Hop Interface
0.0.0.0/0 10.10.10.2 Serial1
10.10.10.0/30 attached Serial1
• Routers need NetFlow-enabled
323232© 2003, Cisco Systems, Inc. All rights reserved.SEC-20088127_05_2003_c1
Source Interface
Flow info Summary
router_A#sh ip cache flow IP packet size distribution (85435 total packets):
1-32 64 96 128 160 192 224 256 288 320 352 384 416 448 480.000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000
512 544 576 1024 1536 2048 2560 3072 3584 4096 4608.000 .000 .000 .000 1.00 .000 .000 .000 .000 .000 .000
IP Flow Switching Cache, 278544 bytes2728 active, 1368 inactive, 85310 added463824 ager polls, 0 flow alloc failuresActive flows timeout in 30 minutesInactive flows timeout in 15 secondslast clearing of statistics never
Protocol Total Flows Packets Bytes Packets Active(Sec) Idle(Sec)-------- Flows /Sec /Flow /Pkt /Sec /Flow /FlowTCP-X 2 0.0 1 1440 0.0 0.0 9.5TCP-other 82580 11.2 1 1440 11.2 0.0 12.0Total: 82582 11.2 1 1440 11.2 0.0 12.0
SrcIf SrcIPaddress DstIf DstIPaddress Pr SrcP DstP PktsEt0/0 132.122.25.60 Se0/0 192.168.1.1 06 9AEE 0007 1 Et0/0 139.57.220.28 Se0/0 192.168.1.1 06 708D 0007 1 Et0/0 165.172.153.65 Se0/0 192.168.1.1 06 CB46 0007 1
show ip cache flow
SrcIf Et0/0Et0/0Et0/0
Flow Details
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.8172_05_2003_c1
333333© 2003, Cisco Systems, Inc. All rights reserved.SEC-20088127_05_2003_c1
show ip cache verbose flowrouter_A#sh ip cache verbose flow IP packet size distribution (23597 total packets):
1-32 64 96 128 160 192 224 256 288 320 352 384 416 448 480.000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000
512 544 576 1024 1536 2048 2560 3072 3584 4096 4608.000 .000 .000 .000 1.00 .000 .000 .000 .000 .000 .000
IP Flow Switching Cache, 278544 bytes1323 active, 2773 inactive, 23533 added151644 ager polls, 0 flow alloc failuresActive flows timeout in 30 minutesInactive flows timeout in 15 secondslast clearing of statistics never
Protocol Total Flows Packets Bytes Packets Active(Sec) Idle(Sec)-------- Flows /Sec /Flow /Pkt /Sec /Flow /FlowTCP-other 22210 3.1 1 1440 3.1 0.0 12.9Total: 22210 3.1 1 1440 3.1 0.0 12.9
SrcIf SrcIPaddress DstIf DstIPaddress Pr TOS Flgs PktsPort Msk AS Port Msk AS NextHop B/Pk ActiveEt0/0 216.120.112.114 Se0/0 192.168.1.1 06 00 10 1 5FA7 /0 0 0007 /0 0 0.0.0.0 1440 0.0Et0/0 175.182.253.65 Se0/0 192.168.1.1 06 00 10 1
Port Msk AS Port Msk AS NextHop B/Pk Active
343434© 2003, Cisco Systems, Inc. All rights reserved.SEC-20088127_05_2003_c1
Tracing Back with ACLs
• Create ACL:access-list 101 permit ip any <target> log-input
• Apply to interface for a few seconds:interface xxx ip access-group 101 in
(wait a few seconds)no ip access-group 101
• Log shows interface the attack comes from
src Interfacemac Address
14:17:21: %SEC-6-IPACCESSLOGP: list 101 permitted tcp 105.12.73.84(0) (FastEthernet0/0 0006.d780.2380) -> 192.168.1.1(0), 1 packet
14:17:22: %SEC-6-IPACCESSLOGP: list 101 permitted tcp 166.159.237.65(0) (FastEthernet0/0 0006.d780.2380) -> 192.168.1.1(0), 1 packet
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.8172_05_2003_c1
353535© 2003, Cisco Systems, Inc. All rights reserved.SEC-20088127_05_2003_c1
Tracing Back with ACLs (Again, Bigger)
Source Interface
MAC Address of Upstream Router
…14:17:21: %SEC-6-IPACCESSLOGP: list 101 permitted tcp
105.12.73.84(0) (FastEthernet0/0 0006.d780.2380) -> 192.168.1.1(0), 1 packet
14:17:22: %SEC-6-IPACCESSLOGP: list 101 permitted tcp 166.159.237.65(0) (FastEthernet0/0 0006.d780.2380) -> 192.168.1.1(0), 1 packet
…
Note: ACL with “log” Does Not Show the Source Interface, “log-input” Does (See above)
363636© 2003, Cisco Systems, Inc. All rights reserved.SEC-20088127_05_2003_c1
Originating Router
Tracing Back across an IXP (…or Any Other Shared Medium)
• NetFlow: Shows i/f only Useless if IXP: Lots of routers behind…
• ACLs with log-input: Shows also the MAC address of the router:
1d00h: %SEC-6-IPACCESSLOGDP: list 101 denied icmp 11.1.1.18 (Ethernet0 0001.96e6.7641) -> 10.1.2.1 (0/0), 169 packets
router#sh arp | include 0001.96e6.7641
Internet 12.1.1.99 152 0001.96e6.7641 ARPA Ethernet0
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.8172_05_2003_c1
373737© 2003, Cisco Systems, Inc. All rights reserved.SEC-20088127_05_2003_c1
Tip for Logging
• Do not log to console
Slow connection (9.6 kbit/s)
Lots of logging à You lose the console
• Logging bufferedAvoids console overload
Automatically wraps
383838© 2003, Cisco Systems, Inc. All rights reserved.SEC-20088127_05_2003_c1
Trace-Back in One Step: ICMP Backscatter
• Border routers: Allow ICMP (rate limited)
• From sink hole router:iBGP update to all ingress routers: “drop all traffic to <victim>” (details later)
• All ingress router drop traffic to <victim>
• And send ICMP unreachables to source!!
• For spoofed sources:Sink hole router logs the ICMPs!
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.8172_05_2003_c1
393939© 2003, Cisco Systems, Inc. All rights reserved.SEC-20088127_05_2003_c1
iBGP Updates: “Drop Packets to Target”
Target
IngressRouters
OtherISPs
Trace-Back in One Step: ICMP Backscatter
Random Sources
Random Sources
Random Sources
Random Sources
Sink Hole Router
404040© 2003, Cisco Systems, Inc. All rights reserved.SEC-20088127_05_2003_c1
Target
Trace-Back in One Step: ICMP Backscatter
Sink Hole Router with Logging
iBGP Updates: “Drop Packets to Target”
ICMP Unreachables
IngressRouters
OtherISPs
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.8172_05_2003_c1
414141© 2003, Cisco Systems, Inc. All rights reserved.SEC-20088127_05_2003_c1
How to Detect Drops on a Router
• Drops can be seen:NetFlow: Destination “null0”
Interface “null0” counters, outbound (simple!)
Also SNMP pollable!
ICMP Unreachables
IngressRouters
OtherISPs
IngressRouters
OtherISPs
424242© 2003, Cisco Systems, Inc. All rights reserved.SEC-20088127_05_2003_c1
ICMP Backscatter
On Sink Hole Router:• Static routes for 1/8,2/8,5/8 (will attract 3/256 of packets)• access-list 105 permit icmp any any log-input
access-list 105 permit ip any any
• Border router sends ICMP unreachable for deleted packets, to source
• If source is random, some will go to 1/8, 2/8, 5/8, …
03:17:22: %SEC-6-IPACCESSLOGDP: list 105 permitted icmp 192.168.0.2 (Serial0/0 *HDLC*) -> 5.52.203.66 (0/0), 1 packet
03:17:38: %SEC-6-IPACCESSLOGDP: list 105 permitted icmp 192.168.0.2 (Serial0/0 *HDLC*) -> 1.167.111.47 (0/0), 1 packet
03:17:52: %SEC-6-IPACCESSLOGDP: list 105 permitted icmp 192.171.12.5 (Serial0/1 *HDLC*) -> 2.153.59.34 (0/0), 1 packet
…
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.8172_05_2003_c1
434343© 2003, Cisco Systems, Inc. All rights reserved.SEC-20088127_05_2003_c1
Summary Tracing DoS Attacks
• Non-spoofed: Technically trivial (IRR)But: Potentially tracing 100’s of sources…
• Spoofed:IP Source Tracker: router by router
NetFlow: Trivial if mechanisms are installedManually: Router by router
ACLs: Has performance impact on some platformsMostly manual: Router by routerBackscatter technique:One step, fast, only for spoofed sources
44© 2003, Cisco Systems, Inc. All rights reserved.SEC-20088127_05_2003_c1
Containing DoS Attacks
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.8172_05_2003_c1
454545© 2003, Cisco Systems, Inc. All rights reserved.SEC-20088127_05_2003_c1
i/f 1i/f 2
i/f 3i/f 1
i/f 2
i/f 3
FIB:. . . S -> i/f x. . .
S D dataS D data
FIB:. . . . . .. . .
S D dataS D data
Any i/f:Forward
Not in FIBor route à null0:
Drop
?
Loose uRPF Check(Unicast Reverse Path Forwarding)
router(config-if)# ip verify unicast source reachable-via any
464646© 2003, Cisco Systems, Inc. All rights reserved.SEC-20088127_05_2003_c1
i/f 1
i/f 2
i/f 3
FIB:. . . 1.2.3.4 à Null0. . .
Not in FIBor route à null0:
Drop
Dropping Traffic from a Source Address
• Goal: Drop all packets from 1.2.3.4
• Static route:1.2.3.4 à Null0
• Loose uRPF: “reachable-via any”
• Minimal CPU impact (2-3%), CEF-based
• Alternative to ACL
uRPF(Loose)
S D dataS D data
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.8172_05_2003_c1
474747© 2003, Cisco Systems, Inc. All rights reserved.SEC-20088127_05_2003_c1
Making It Scalable
• Problem: Potentially 100’s of sources to track and shun
• Manually: For few sources only
• On big ISP networks:
Scalable mechanisms required!
Idea: Use routing to distribute information
484848© 2003, Cisco Systems, Inc. All rights reserved.SEC-20088127_05_2003_c1
Shunning with uRPF and BGP
• ip route x.x.x.x null0 is manual :-(
• BGP cannot send “next-hop null0”… but:
• BGP can send “next-hop 192.0.2.1”
• And on each border router:ip route 192.0.2.1 null0
• Router receives iBGP routing update:“Route x.x.x.x next-hop 192.0.2.1” (comm: local-AS)
And it has an ip route 192.0.2.1 null0
Thus: x.x.x.x à null0 (note: CEF required!)
• With uRPF: Source x.x.x.x also à null0
Trick:
Not in Use!
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.8172_05_2003_c1
494949© 2003, Cisco Systems, Inc. All rights reserved.SEC-20088127_05_2003_c1
Effect of BGP Remote Trigger
• Traffic to/from a specific subnet will be sent to null0
• Automatically, on all border routers
• No attack traffic on backbone
• But…where is the attack coming from??? which upstream ISPs to notify???
505050© 2003, Cisco Systems, Inc. All rights reserved.SEC-20088127_05_2003_c1
ICMP Backscatter
On Sink Hole Router:• Static routes for 1/8,2/8,5/8 (will attract 3/256 of packets)• access-list 105 permit icmp any any log-input access-list
105 permit ip any any
• Border router sends ICMP unreachable for deleted packets, to source
• If source is random, some will go to 1/8, 2/8, 5/8, …
03:17:22: %SEC-6-IPACCESSLOGDP: list 105 permitted icmp 192.168.0.2 (Serial0/0 *HDLC*) -> 5.52.203.66 (0/0), 1 packet
03:17:38: %SEC-6-IPACCESSLOGDP: list 105 permitted icmp 192.168.0.2 (Serial0/0 *HDLC*) -> 1.167.111.47 (0/0), 1 packet
03:17:52: %SEC-6-IPACCESSLOGDP: list 105 permitted icmp 192.171.12.5 (Serial0/1 *HDLC*) -> 2.153.59.34 (0/0), 1 packet
…
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.8172_05_2003_c1
515151© 2003, Cisco Systems, Inc. All rights reserved.SEC-20088127_05_2003_c1
Using CAR to Rate Limit Attack Traffic
• Other ACLs for other attacks:UDP-based attacks…
• Watch your CPU!!!
Limit to 3 Mbit/sinterface xy
rate-limit output access-group 2020 3000000512000 786000 conform-action transmit
exceed-action drop
access-list 2020 permit icmp any any echo-reply
525252© 2003, Cisco Systems, Inc. All rights reserved.SEC-20088127_05_2003_c1
Using CAR against SYN Flood Attacks?
• Same rate limiting, with this ACL:access-list 169 permit tcp any host victim-host syn
access-list 169 deny ip any any
• Limits ALL syns, also “legal” ones
• But: Blocks only when under DoS!
Effectively Like Blocking All (under DoS)Effectively Like Blocking All (under DoS)
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.8172_05_2003_c1
535353© 2003, Cisco Systems, Inc. All rights reserved.SEC-20088127_05_2003_c1
Important!
Stopping SYN Attacks: TCP Intercept
• SYN rate-limiting: Kills “good” traffic, too• Proper solution: TCP intercept• Devices where implemented:
Routers/Cisco IOSPIX (Firewall) CSS (Content Switches)FWSM (Firewall Service Module, Cat6k)CSM (Content Switching Module, Cat6k)
• DoS against Web? à content story!!!
High Performance!High Performance!
545454© 2003, Cisco Systems, Inc. All rights reserved.SEC-20088127_05_2003_c1
Configuring CAR through BGP
• QPPB (QoS Policy Propagation with BGP)
• On each border router: Define a CAR policy, linked to a QoS group (normally unused)
• To limit an attack, assign the network to the QoS group (BGP community)
See: http://www.cisco.com/univercd/cc/td/doc/product/software/ios111/cc111/bgpprop.htm
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.8172_05_2003_c1
555555© 2003, Cisco Systems, Inc. All rights reserved.SEC-20088127_05_2003_c1
Summary: Containing DoS Attacks
• ACLs:
Manual, performance impact
• uRPF:
Stops non-existing sources
Automated with BGP for specific shunning
• CAR: Limit attack flow, performance impact
Manual or automated via QPPB (BGP)
565656© 2003, Cisco Systems, Inc. All rights reserved.SEC-20088127_05_2003_c1
Reality Check: Does All This Make Aense?
• Rate limit:Mostly we also limit “good” traffic
à Users are also limited
Exceptions: ICMP, maybe UDP
• ACLs and Null0:We drop all traffic to a server
Also “good” traffic
Goal: ACLs as specific as possible
We Need to We Need to Become Become
“Smarter”!!“Smarter”!!
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.8172_05_2003_c1
57© 2003, Cisco Systems, Inc. All rights reserved.SEC-20088127_05_2003_c1
Additional Solutions:Arbor and Riverhead
Cisco AVVID Partner Program
585858© 2003, Cisco Systems, Inc. All rights reserved.SEC-20088127_05_2003_c1
The Need for More…
• On today’s routers you have limited CPUMore “intelligence” needed
• Blocking, rate-limit mostly too course
Also affects “legal” traffic
Need a Way to Separate Need a Way to Separate “Good” from “Bad” Packets“Good” from “Bad” Packets
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.8172_05_2003_c1
595959© 2003, Cisco Systems, Inc. All rights reserved.SEC-20088127_05_2003_c1
Arbor NetworksPeakflow DoS: Hardware
• 1RU rack height (1.7”)
• Collects flow statistics from border and edge routers
• Builds dynamic traffic baseline to detect bandwidth anomalies
• Works with Cisco routers and IDS/firewalls
• 2RU rack height (3.3”),
• Aggregates distilled anomaly data from collectors
• Correlates distributed events to create network-wide view of all DoS attacks
CollectorCollector
ControllerController
606060© 2003, Cisco Systems, Inc. All rights reserved.SEC-20088127_05_2003_c1
Arbor Networks: SP Solution
Service Provider AService Provider B
Service Provider C
Customer Web Server
IDS
Firewall
Service Provider AService Provider B
Service Provider C
Customer Web Server
IDS
Firewall
Service Provider AService Provider B
Service Provider C
Customer Web Server
IDS
Firewall
5.Filter: Recommends Filters (X)
2.Monitor: Analyze Traffic for Anomalies1. Profile: Base Line Traffic Patterns in the Network
4.Trace: Trace the Attack to Its Source 3.Detect: Forward Anomaly Fingerprints to Controllers
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.8172_05_2003_c1
616161© 2003, Cisco Systems, Inc. All rights reserved.SEC-20088127_05_2003_c1
Database
MaliciousPackets
Riverhead: Basic Concepts
1. Detection
2. Diversion of victims’ traffic
3. Sieve out malicious traffic
4. Legitimate traffic continues on its route
Victim Traffic
RVictim
Clean Traffic
62© 2003, Cisco Systems, Inc. All rights reserved.SEC-20088127_05_2003_c1
Wrap Up
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.8172_05_2003_c1
636363© 2003, Cisco Systems, Inc. All rights reserved.SEC-20088127_05_2003_c1
What We Can Do Now
• Detect DoS Attacks (SNMP, NetFlow, ACL)
• Trace back random packet floods (NetFlow, ACLs, IP source tracker)
• Shun a source (uRFP, ACL)
• Shun a destination (routing, ACL)
• Limit attacking traffic (CAR, PIRC)
• Remote trigger via iBPG
• Protect web servers (CSM/Content Networking)
• Understand partner solutions (Arbor, Riverhead)
646464© 2003, Cisco Systems, Inc. All rights reserved.SEC-20088127_05_2003_c1
• A good DoS attack may not be stoppable
• Key question: Is a packet “good” or “bad”?
Current solution: Drop all…
à You’re completing the DoS attack!!!
• For Web: Content story seems the only true solution!
• Other: TCP-intercept (SYN attacks)
CSM/FWSM have the performance needed!! (tens of thousands of pps)
DoS: When Routers Alone Can’t Help
Important!
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.8172_05_2003_c1
656565© 2003, Cisco Systems, Inc. All rights reserved.SEC-20088127_05_2003_c1
Some Loose Ends…
• ACLs with “log(-input)”: Nice, but CPU intensive!
• Debug ip packet: Your CPU hates you!!!
Always use with ACL
• ip route x.x.x.x null0: Configure ICMP rate-limiting or no ip unreachables!
• Routing security (MD5 auth): Push ISP to do it!!! (more attacks on routing will come)
666666© 2003, Cisco Systems, Inc. All rights reserved.SEC-20088127_05_2003_c1
Tip: Scheduler Allocate
• Schedules CPU time spent on processes versus interrupts
Very Useful under Heavy Load!Very Useful under Heavy Load!Recommended Standard Config!Recommended Standard Config!
Syntax:
scheduler allocate <interrupt> <processes>
<interrupt>: 3000-60000 Microseconds Handling Networkinterrupts
<processes>: 1000-8000 Microseconds Running Processes
Example:
router(config)#scheduler allocate 8000 8000
Default Values: 4000 200
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.8172_05_2003_c1
676767© 2003, Cisco Systems, Inc. All rights reserved.SEC-20088127_05_2003_c1
Recommendations for ISPs
• Preventative measures: ACLs, uRPF, CAR…See ISP Essentials
• Monitor your routers and alarm on:CPU, line load, memory…
• Use NetFlow plus collector s/w:Usage statistics, DoS detection, DoS tracing through the network
• Be prepared:Technically: Understand the routers
Operationally: Have procedures in place, know your upstream/downstream contacts, have a CERT
686868© 2003, Cisco Systems, Inc. All rights reserved.SEC-20088127_05_2003_c1
What Will the Future Bring?
• More PCs always online (DSL, Cable)
The vulnerabilities are here!
• More vulnerabilities and zombies?
• IDS systems triggering routing updates?
• More “legal” traffic?
• Need to distinguish “good” from “bad” packets
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.8172_05_2003_c1
696969© 2003, Cisco Systems, Inc. All rights reserved.SEC-20088127_05_2003_c1
“The first day of summer vacation is a DoS attack against the highway system.”
Other Types of DoS :-)
(What Can We Do If Future Dos Attacks Use Many PCs to Send Millions of Perfectly Legal HTTP Request?)
70© 2003, Cisco Systems, Inc. All rights reserved.SEC-20088127_05_2003_c1
Questions?
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.8172_05_2003_c1
717171© 2003, Cisco Systems, Inc. All rights reserved.SEC-20088127_05_2003_c1
References (Non-Cisco)
• DoS Detection: “Tackling Network DoS on Transit Networks”: David Harmelin, DANTE, March 2001 (describes a detection method based on netflow)[http://www.dante.net/pubs/dip/42/42.html]“Inferring Internet Denial-of-Service Activity”: David Moore et al, May 2001; (described a new method to detect DoS attacks, based on the return traffic from the victims, analysed on a /8 network; very interesting reading) [http://www.caida.org/outreach/papers/backscatter/index.xml]“The spread of the code red worm”: David Moore, CAIDA, July 2001(using the above to detect how this worm spread across the Internet) [http://www.caida.org/analysis/security/code-red/]
• DoS Tracing:“Tracing Spoofed IP Addresses”: Rob Thomas, Feb 2001; (good technical description of using netflow to trace back a flow)[http://www.enteract.com/~robt/Docs/Articles/tracking-spoofed.html]
• Other: “DoS attacks against GRC.com”: Steve Gibson, GRC, June 2001 (a real life description of attacks from the victim side; somewhat disputed, but fun to read!) [http://grc.com/dos/grcdos.htm]
727272© 2003, Cisco Systems, Inc. All rights reserved.SEC-20088127_05_2003_c1
References (Cisco)
• Product Security:Cisco’s Product Vulnerabilities; A page that every SE MUST know!!! [http://www.cisco.com/warp/public/707/advisory.html]Security Reference Information: Various white papers on DoS attacks and how to defeat them [http://www.cisco.com/warp/public/707/ref.html]
• ISP Essentials:Technical tips for ISPs every ISP should know[ftp://ftp-eng.cisco.com/cons/isp/]
• Technical tips:Troubleshooting High CPU Utilization on Cisco Routers [http://www.cisco.com/warp/public/63/highcpu.html]The “show processes” command[http://www.cisco.com/warp/public/63/showproc_cpu.html]NetFlow Performance White Paper[http://www.cisco.com/warp/public/cc/pd/iosw/prodlit/ntfo_wp.htm]
• Mailing lists:Cust-security-announce: All customers should be on this list Cust-security-discuss: For informal discussions
Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.8172_05_2003_c1
737373© 2003, Cisco Systems, Inc. All rights reserved.SEC-20088127_05_2003_c1
Recommended Reading
CCIE Security Exam Certification GuideISBN: 1587200651
CCIE Practical Studies: SecurityISBN: 1587051109
Network Security Principles and PracticesISBN: 1587050250
Available on-site at the Cisco Company Store
747474© 2003, Cisco Systems, Inc. All rights reserved.SEC-20088127_05_2003_c1
Please Complete Your Evaluation Form
Session SEC-2008