HOW YOU COULD HAVE SURVIVED

THE AWS CHRISTMAS EVE OUTAGE

ARCHITECTING FOR FAILURES IN THE

CLOUD

Pavan Verma

Founder, P3 InfoTech Solutions Pvt. Ltd.

pavan@p3infotech.in, @YingYangPavan

1

How are datacenter failures

relevant to me?

2

Types of Failures in a Datacenter

• Electronic components – CPUs, Memory

• Mechanical components – Hard Disks, Fans

• Electrical components – Power Supplies, Air

conditioners

• Networking equipment – Network cables,

Routers, Switches

• Software bugs

• Power disruption

• Human errors

3

Cost of Failures

• Tangible cost

• Lost business = Business volume (Revenues) /

Duration of failure

• Cost of lost data or Time to re-create lost data

• Intangible cost

• Reputation

• Frustration

• Lost business opportunity

4

Techniques to deal with failures

5

Backup

• Backup = Copy of the data from a time

before the failure

• Can restore data after the failure to a state

from before the failure

• Limits the extent of data loss during a

failure

• Types of Backup

• Disk-to-tape – Offline, Slower restore, Cheaper

• Disk-to-disk – Online, Faster restore, Costlier

6

Backup in AWS

• Snapshots for EBS volumes

• Database backups with RDS

• Redundant copies of S3 objects (*)

7

High Availability (HA)

• Ability of the application to service requests

in spite of failure of some components

• Most prevalent notion of High Availability

• Ability to application to tolerate single

component failures

• Application has no single point of failure

• How is high availability achieved

• Redundant components

• Switchover traffic from failed component to

working component

8

High Availability (2)

• Two types of redundant components

• Active-Active

• Active-Passive

• Examples

• Power Supplies

• Servers

• Databases

9

High Availability in AWS

• Availability Zones (AZ)

• Elastic Load Balancer

• Database replicas

10

User

Auto Scaling Group of

EC2 Instances

EC2 EC2

S3

ELB

Auto Scaling Group of

EC2 Instances

EC2 EC2

RDS

Slave

RDS

Master

AZ #1 AZ #2

Reference Architecture for High

Availability setup in AWS

11

Disaster Recovery

• Ability to resume operations after a disaster

• How bad can a disaster be?

• Entire datacenter may be destroyed or become

inoperational

• Examples: 9/11, Hurricane Sandy, Northeast

blackout of 2003

• Affects all Availability Zones in a Region

12

Disaster Recovery Solutions

• Disaster recovery solutions involve combination of

• Replication / Backup of data to a different geography [On-going]

• Start operations from the DR site [when disaster occurs]

• Switch-over traffic to DR site

• Sync data and restore operations to primary site once it becomes operational again

• Since DR involves a different Geo, data replication/backup happens over WAN

13

Recovery Point and Recovery Time

• Recovery Point (RP) = Duration of time for

which data is lost

• Recovery Time (RT) = Duration of time in

which the application is restored

• Low numbers are better for both RP and RT

• Often framed as RPO and RTO as part of

business continuity planning

14

Recovery Point and Recovery Time

• Backup = High RP and High RT

• High Availability = Zero RP and Zero RT

• Disaster Recovery = RP and RT between

Backup and HA

15

Conclusion

• Topic of IT failures is very relevant for business operations

• Key issues with failures

• Unavailability of application

• Loss of data

• Techniques to handle failures – Backups, High Availability, Disaster Recovery

• AWS provides mechanisms to deal with failures

• Recovery Point and Recovery Time

16

17

Have a great Barcamp Bangalore 2013!