ddos resiliency with amazon web services (sec305) | aws re:invent 2013

© 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.

DDoS Resilience with Amazon Web Services

[email protected]

November 14, 2013

Agenda

• Anatomy of DDoS

• Things We Do So You Don’t Have To

• Designing for Availability

• Attack Response

DDoS Facts

• Yes, DDoS attacks are on the rise and the big

ones are getting bigger

• …although those attacks average out to

~14Gbps* and target services owners ~1 per

year

*source: Arbor Networks

DDoS Facts

*source: Arbor Networks

Percentile Max Gbps Duration

(minutes) 10 2.39 5.87

20 4.28 7.68

30 6.55 9.00

40 8.27 10.53

50 10.49 13.23

60 11.85 16.80

70 13.97 23.12

80 17.38 35.87

90 25.45 66.13

95 35.74 141.74

99 84.90 906.80

Max 299.43

Average 13.81

DDoS Anatomy

Application Exhaustion

/search.php?expensive-params

attacker service

DDoS Anatomy

Host Exhaustion

attacker

attacker

service

DDoS Anatomy

Traditional Datacenter Exhaustion

attacker

traditional

datacenter transit

attacker

attacker

DDoS Anatomy

Intermediary Exhaustion

attacke

r

traditional

datacenter transit

transit

transit

transit

attacke

r

attacke

r

attacke

r

attacke

r

attacke

r

attacke

r

DDoS Anatomy

• Large enough attacks consume the capacity of

application layer, host, datacenter connectivity,

Internet connectivity, or intermediary networks

How can we help you?

• Scale and Diversity of AWS

• Resilient Service Designs

• Business or Enterprise Support

Things We Do So You Don’t Have To

Scale

model credit:

Scale

traditional

datacenter transit

AWS

region

Scale

transit

transit

transit

More Bandwidth

Scale

transit AWS

region

transit

transit

More Compute

Scale

transit AWS

region

AWS

edge

AWS

edge

AWS

edge transit

transit

More Points of Presence

Scale Attack Absorbed

transit

attacker

attacker

attacker

AWS

region

AWS

edge

AWS

edge

AWS

edge transit

transit

Diversity

Internet Transit and Peering

AWS

region peer

transit

peer

peer

peer

transit

transit

peer

Diversity

Diversity

Amazon Route 53 Example - Anycast Striping

• Leverages Resolver Behavior

• Edge Location Diversity

• Network Path Diversity

Delegation Set [nated@xyz ~]$ dig NS internetkitties.com

;; QUESTION SECTION:

;internetkitties.com. IN NS

;; ANSWER SECTION:

internetkitties.com. 172800 IN NS ns-1131.awsdns-13.org.

internetkitties.com. 172800 IN NS ns-1751.awsdns-26.co.uk.

internetkitties.com. 172800 IN NS ns-340.awsdns-42.com.

internetkitties.com. 172800 IN NS ns-952.awsdns-55.net.

awsdns-13.org.

awsdns-26.co.uk.

awsdns-42.com.

awsdns-55.net.

Edge Location Diversity

Network Path Diversity

awsdns-13.org.

awsdns-26.co.uk.

awsdns-42.com.

awsdns-55.net.

[nated@xyz ~]$ traceroute ns-1131.awsdns-13.org.

traceroute to ns-1131.awsdns-13.org (205.251.196.107), 64 hops max, 52 byte packets

1 (192.168.1.1) 1.748 ms 0.830 ms 0.750 ms

2 * * *

3 cat.seattle.wa.seattle.comcast.net (68.85.255.255) 14.634 ms 12.822 ms 10.774 ms

4 ae-20-0-ar03.burien.wa.seattle.comcast.net (69.139.164.125) 31.766 ms 13.898 ms

5 ae-20-0-ar03.seattle.wa.seattle.comcast.net (69.139.164.129) 20.108 ms

6 he-1-7-0-0-11-cr01.seattle.wa.ibone.comcast.net (68.86.93.5) 18.781 ms

7 ae12.edge2.seattle3.level3.net (4.68.63.65) 34.371 ms 36.504 ms 27.301 ms

8 ae-31-51.ebr1.seattle1.level3.net (4.69.147.150) 48.557 ms 60.610 ms 56.751 ms

9 ae-7-7.ebr2.sanjose1.level3.net (4.69.132.49) 58.662 ms 46.830 ms 62.458 ms


11 ae-6-6.ebr2.losangeles1.level3.net (4.69.148.201) 55.190 ms 58.829 ms 55.751 ms

12 ae-92-92.csw4.losangeles1.level3.net (4.69.137.30) 49.261 ms

13 ae-3-80.edge5.losangeles1.level3.net (4.69.144.139) 58.707 ms 53.091 ms

14 amazon.com.edge5.losangeles1.level3.net (205.129.4.26) 46.477 ms 36.525 ms 42.110 ms

15 LAX3

[nated@xyz ~]$ traceroute ns-1751.awsdns-26.co.uk.

traceroute to ns-1751.awsdns-26.co.uk (205.251.198.215), 64 hops max, 52 byte packets

1 (192.168.1.1) 1.298 ms 0.755 ms 0.694 ms

2 * * *


4 ae-20-0-ar03.seattle.wa.seattle.comcast.net (69.139.164.129) 17.281 ms 18.580 ms 17.906





9 ae-62-62.csw1.sanjose1.level3.net (4.69.153.18) 44.181 ms

10 ae-3-80.edge1.sanjose3.level3.net (4.69.152.144) 46.817 ms

11 4.53.208.22 (4.53.208.22) 54.634 ms 60.111 ms 44.187 ms

12 205.251.229.155 (205.251.229.155) 47.758 ms

13 205.251.230.91 (205.251.230.91) 52.714 ms 43.560 ms

14 SFO5

[nated@xyz ~]$ traceroute ns-340.awsdns-42.com.

traceroute to ns-340.awsdns-42.com (205.251.193.84), 64 hops max, 52 byte packets

1 (192.168.1.1) 2.444 ms 1.676 ms 1.028 ms

2 * * *


4 ae-20-0-ar03.seattle.wa.seattle.comcast.net (69.139.164.129) 24.366 ms 20.753 ms 29.955 ms




8 ae-2-2.ebr2.denver1.level3.net (4.69.132.54) 163.723 ms 159.037 ms 174.670 ms

9 ae-3-3.ebr1.chicago2.level3.net (4.69.132.62) 169.379 ms 167.307 ms 168.454 ms

10 ae-6-6.ebr1.chicago1.level3.net (4.69.140.189) 166.002 ms 168.125 ms 164.232 ms

11 ae-2-2.ebr2.newyork2.level3.net (4.69.132.66) 167.861 ms 167.893 ms 160.681 ms

12 ae-1-100.ebr1.newyork2.level3.net (4.69.135.253) 163.919 ms 166.782 ms 161.686 ms

13 4.69.201.45 (4.69.201.45) 164.023 ms

14 ae-42-42.ebr2.london1.level3.net (4.69.137.69) 165.560 ms 160.461 ms

15 ae-46-46.ebr2.amsterdam1.level3.net (4.69.143.73) 165.627 ms

16 ae-59-224.csw2.amsterdam1.level3.ne (t4.69.153.214) 172.909 ms 166.052 ms

17 4.69.162.154 (4.69.162.154) 166.353 ms

18 212.72.41.162 (212.72.41.162) 171.714 ms 174.033 ms 179.219 ms

19 AMS50

[nated@xyz ~]$ traceroute ns-952.awsdns-55.net.

traceroute to ns-952.awsdns-55.net (205.251.195.184), 64 hops max, 52 byte packets

1 (192.168.1.1) 1.352 ms 0.642 ms 0.630 ms

2 * * *


4 be-1-ur08.seattle.wa.seattle.comcast.net (69.139.164.134) 13.561 ms

5 ae-1-0-ar03.seattle.wa.seattle.comcast.net (68.85.240.94) 21.009 ms

6 he-1-12-0-0-11-cr01.seattle.wa.ibone.comcast.net (68.86.93.177) 17.366 ms 19.162 ms

7 be-12-pe03.seattle.wa.ibone.comcast.net (68.86.84.106) 19.949 ms 22.968 ms 24.976 ms

8 * * *

9 * * *

10 * 65-122-235-178.dia.static.qwest.net (65.122.235.178) 40.707 ms 30.916 ms

11 205.251.225.22 (205.251.225.22) 85.275 ms

12 205.251.225.122 (205.251.225.122) 35.017 ms 38.568 ms

13 205.251.226.136 (205.251.226.136) 36.560 ms

14 SEA50

Striping in Action

awsdns-13.org.

awsdns-26.co.uk.

awsdns-42.com.

awsdns-55.net.

Striping in Action

Diversity

transit

attacker

AWS

region

AWS

edge

AWS

edge

AWS

edge

client

client

AWS

edge

Diversity

• Amazon Route 53 - Anycast Striping

• Amazon CloudFront Edge Locations

• AWS Regions


• Scale and Diversity of AWS




• Amazon Route 53 and Amazon CloudFront



Designing for Resilience


• N+1 Failover

• Resilient Clients

• Capped Workloads

• Process Isolation

• Shuffle Sharding

N+1 Failover

• Scale Out, Plus Redundancy

N+1 Failover


• Failure of 1/100 < Failure of 1/10

N+1 Failover


• Failure of 1/100 < Failure of 1/10

• Automatic Failover with Health Checked DNS

N+1 Failover

client

attacker

N+1 Failover

Check out Amazon Route 53

Health Checks


• N+1 Failover





Resilient Clients

• Use multi-record RRSets

• Randomize the record on connect retry

• Popular HTTP clients already do this!

Resilient Clients [nated@xyz ~]$ dig www.internetkitties.com


;www.internetkitties.com. IN A

;; ANSWER SECTION:

www.internetkitties.com. 32 IN CNAME d3g5kqnbrlf3fg.cloudfront.net.

d3g5kqnbrlf3fg.cloudfront.net. 30 IN A 54.230.69.190








Resilient Clients

Num Time Source Destination

4 2.535515 10.61.60.17 54.230.69.141 [SYN]

5 2.736659 10.61.60.17 54.230.69.190 [SYN]

6 2.93782 10.61.60.17 54.230.71.141 [SYN]

7 3.138996 10.61.60.17 54.230.71.172 [SYN]

8 3.339767 10.61.60.17 54.230.71.233 [SYN]

9 3.540963 10.61.60.17 54.240.188.66 [SYN]

11 3.541123 10.61.60.17 54.230.68.41 [SYN]

12 3.742296 10.61.60.17 54.230.68.212 [SYN]

13 3.824502 10.61.60.17 54.230.69.190 [SYN]

14 3.824515 10.61.60.17 54.230.69.141 [SYN]

15 4.024809 10.61.60.17 54.230.71.141 [SYN]

16 4.225094 10.61.60.17 54.230.71.172 [SYN]

Browser Packet Capture

Client Retry Behavior, SYN Timeout Browser OS Rotates

IPs

Time to

Rotation

Chrome 30.0.1599 Windows 7 Yes 12

Internet Explorer 8 Windows 7 Yes 12

Firefox 25 Windows 7 Yes 20

Safari 5.0.5 Windows 7 Yes 20

Safari 6.0.5 OSX 10.7.5 Yes <1

Firefox 25 OSX 10.7.5

Yes (2) <1

Chrome 32.0.1678 OSX 10.7.5

Yes (2) DNS TTL, or

Refresh

Resilient Clients attacker

service

client

Resilient Clients


• N+1 Failover





Capped Workloads

• Protect Application Layer Capacity

• Strive for Sameness

• Throttle or Sample Request Workloads

Strive for Sameness

Application Exhaustion


attacker service

Strive for Sameness


attacker service

Search_Result_Page_1

Capped Workloads Host/OS

~500K to 5M pps

AppLayer

~1K to ~10K rps


~500K to 5M pps

AppLayer

~1K to ~10K rps

Auth

Core

Logging

DAL


~500K to 5M pps

AppLayer

~1K to ~10K rps

Auth

Core

Logging

DAL

Throttle

~10 to ~100K rps


~500K to 5M pps

AppLayer

~1K to ~10K rps

Auth

Core

Logging

DAL

Throttle

~10 to ~100K rps

1,000 samples /

sec


• N+1 Failover





Process Isolation

• Isolate application components across

processes

• Let the OS protect critical resources

Process Isolation

Auth

Core

Logging

DAL


• N+1 Failover





Evolution of Resilience

client

client

N Choose M Isolation

• 2 endpoints 2 AZs = 4 permutations



• 8 endpoints 2 AZs = 64

Shuffle Sharding – Amazon Route 53

• Define Availability Lattice • Stripes – Edge Location

• Braids – Host Isolation

• Assign Endpoints to the Lattice • Virtual Name Servers

• Allocate Endpoints to Resources • Hosted Zone Delegate Set

Non-Overlapping Delegation Sets


;gray.internetkitties.com. IN NS

;; ANSWER SECTION:

ns-1131.awsdns-13.org.

ns-1751.awsdns-26.co.uk.

ns-340.awsdns-42.com.

ns-952.awsdns-55.net.


;orange.internetkitties.org. IN NS

;; ANSWER SECTION:





Shuffle Sharding

.com

.net

.co.uk

.org

Shuffle Sharding

.com

.net

.co.uk

.org

A B C D



Shuffle Sharding

.com

.net

.co.uk

.org

A B C D

gray.internetkitties.com

orange.internetkitties.org

Non-Overlapping Delegation Sets


;gray.internetkitties.com. IN NS

;; ANSWER SECTION:






;orange.internetkitties.org. IN NS

;; ANSWER SECTION:





Shuffle Sharding Resilience



.co.uk

.org

client

A

B

C

D

A

B

C

D

attacke

r

Shuffle Sharding Resilience



.co.uk

.org

client

attacke

r

A

B

C

D

A

B

C

D

Shuffle Sharding Toolkit

• Define a Lattice of Availability

• Allocate Service Resources to the Lattice

• Assign Customers Isolated Resources

• https://github.com/awslabs/route53-infima

Lattice Configuration // Create a 1-D lattice with "AvailabilityZone” as the dimension

OneDimensionalLattice<HealthCheckedRecordSet> myServiceLayout =

new OneDimensionalLattice<HealthCheckedRecordSet>("AvailabilityZone”);

Lattice Configuration // Add endpoints in the us-west-1a Availability zone

myServiceLayout.addEndpoint("us-west-1a”,

new HealthCheckedRecordSet("192.0.2.1"));





…

// Add endpoints in the us-west-1b Availability zone

myServiceLayout.addEndpoint("us-west-1b”


…

Shuffle Shard // Create a shuffle sharder

SimpleSignatureShuffleSharder shuffleSharder = new SimpleSignatureShuffleSharder(5353L);

Lattice shard = shuffleSharder.shuffleShard(myServiceLayout, "v123543234", 1);

Vulcanized Lattice // Create a shuffle sharder

SimpleSignatureShuffleSharder shuffleSharder = new SimpleSignatureShuffleSharder(5353L);

Lattice shard = shuffleSharder.shuffleShard(myServiceLayout, "v123543234", 1);

// Create a RubberTree of DNS records

Route53RubberTree rubberTree =

new Route53RubberTree(”v123543234.video.internetkitties.com", shard);

List rrsets = rubberTree.vulcanize();

Lattice Shard RRSet

[nated@xyz ~]$ dig v123543234.video.internetkitties.com


; v123543234.video.internetkitties.com. IN A

;; ANSWER SECTION:

v123543234.video.internetkitties.com. 60 IN A 192.0.2.12



us-west-1b

us-west-1a

us-west-1c


• N+1 Failover





Attack Response

Attack Response

• Detection

• Src-IP Blocking

• Engaging Customer Support

Detect

• Traffic Spikes, Drops

• CPU Utilization

• Network Stats

Detect

• Use Resilience Patterns to Access Logs

• X-Forwarded-For

• Sort and Sum

X-Forwarded-For

• Use a trusted load balancer or proxy

X-Forwarded-For


• Enable logging

X-Forwarded-For


• Enable logging – IIS7

• Install ‘IIS Advanced Logging’

• Configure X-Forwarded-For field

X-Forwarded-For

Enable Logging

nginx:

if($http_x_forwarded_for !='-’) {

log_format main '$http_x_forwarded_for - $remote_user

[$time_local] $status '

'"$request" $body_bytes_sent "$http_referer" '

'"$http_user_agent" "$remote_addr"';

}

else {

log_format main '$remote_addr - $remote_user [$time_local]

$status '

'"$request" $body_bytes_sent "$http_referer" '

'"$http_user_agent" "$http_x_forwarded_for"';

}

X-Forwarded-For


• Enable X-Forwarded-For logging

Sort & Sum

• Used to identify “top talkers”

[[email protected] ~]$ grep 'expensive-param' ./access.log | awk '{print $1}' |

sort | uniq -c | tail

2 10.54.4.1

3 10.63.34.1

5 10.23.97.212

1182 10.54.0.183

Src-IP Blacklisting

• Host-Level Firewalling

• Web-Server Configuration

• VPC Network ACLs

• Web Application Firewall

Src-IP Blacklisting

• Host-Level Firewalling (IPTables)

• Web-Server Configuration (Nginx / Apache, IIS)



Src-IP Blacklisting


• Web-Server Configuration



VPC Network ACLs

• Apply to a VPC subnet

• Supports DENY rules

VPC Network ACLs

• Enter each source IP

• Set DENY

Src-IP Blacklisting




Web Application Firewall

• Src-IP Blacklist

• HTTP Headers (X-Forwarded-For)

• URI-Based Filtering

• Advanced Throttling

Attack Response

• Detection

• Src-IP Blocking

• Engaging Customer Support

Engaging Customer Support

http://aws.amazon.com/premiumsupport/



Summary

How can we help? • Scale and Diversity

• Route 53 and CloudFront

• Business and Enterprise

Support

Resilient Design • Availability Lattice


• N+1 Failover



• Process Isolation Attack Response • Enable X-Forwarded-For Logging

• Detect, Sum and Sort

• Src-IP Blacklist

• Engage Customer Support

Please give us your feedback on this

presentation

As a thank you, we will select prize

winners daily for completed surveys!

SEC305

ddos resiliency with amazon web services (sec305) | aws re:invent 2013

Technology