Storage Solutions in the AWS Cloud

Miles Ward – Enterprise

Solutions Architect

On-Premise Storage Options

SAN – network-attached block devices: LUNs

DAS – local block devices (disks)

NAS – network attached file systems, filers

Differ in, Performance, Durability, and Cost

Often used to form a Storage Hierarchy

Traditional Storage

Cloud Storage

AWS offers multiple cloud-based storage options.

• Amazon S3 (Simple Storage Service)

• Amazon EBS (Elastic Block Storage)

• Amazon EC2 Local Instance Store (Ephemeral volumes)

Each has a unique combination of performance, durability, cost, and interface

Often used to form a Storage Hierarchy

Other Storage options?

Amazon S3, EBS, and EC2 Instance Store volumes are the basic storage building blocks, but for more specialized uses, or managed resources, we have other options as well:

Cloudfront

Amazon RDS

SimpleDB

SQS

Relational Databases on EC2

Storage by the Numbers

S3 EBS Local

Scale Unlimited Storage

5TB objects

100 name spaces (buckets)

1TB per volume

~44TB per server

147 – 1690GB

1-5 volumes

Durability 99.999999999% (standard)

99.99%

(RRS)

.1-.5% AFR Ephemeral design

requires application-

level redundancy

Latency High Low Very Low

Management Zero Maintenance Snapshots, RAID,

Consistency

Manual Redundancy

Locality Region-Centric AZ-Specific In-machine

Interface REST or SOAP

Web Services

Block Device Block Device

S3 (Simple Storage Service)

Pay for exactly what you use

Highly Durable

Limitlessly scalable

Natively On-Line

S3 Best Practices

Replicate Metadata

Parallel IO for max speed

Multipart Upload

Sorted Writes

Resource-level IAM permissions

S3 Use Cases and Users

Media Serving

Media Sharing

File Storage / Backup

Static Content

Big Data

Amazon EBS

Pay for what you provision

POSIX compliant network attached block device

Native Redundancy and Write Cache; optimized for random writes and reads.

EBS Best Practices

Pre-warming

Snapshots

Striping (RAID 0)

Block-device-mapping

Application hosting

Data Processing

Database

Large Data

EBS Use Cases and Users

Envoy Media Group Case Study Storage for multimedia content delivery on Amazon Web Services (AWS)

Gluster - 640 W. California Ave. Suite 200, Sunnyvale, CA 94086 - 1.800.805.5215 - www.gluster.com

Introduction

Established in 2006, Envoy Media Group is an innovative marketing firm

that offers its clients highly-targeted media campaigns across multiple

media channels including television, radio, the Internet, direct video and

with a unique

and powerful advantage. By capitalizing on their multiplemarketing

channels, Envoy caters to selected businesses offering a focused plan and

implementation of concentrated campaigns both online and offine.

Envoy moved their IT infrastructure from a traditional co-location facility to Amazon Web Services to gain greater flexibility, take

advantage of -as-you- e their media assets were always available, easy

to share, and promptly delivered, Envoy selected Gluster in concert with Amazon EC2, EBS volumes, and S3.

Challenges Envoy currently hosts their entire web presence on Amazon Web Services (AWS)

which has helped them meet the needs of their growing business and customer

base. One of their biggest challenges was ensuring the IT infrastructure was able

to respond to the highly dynamic nature of demand such as when campaigns

launch and then grow exponentially. From a data storage standpoint, they also

needed a solution that integrated easily with AWS and provided high availability.

Storing the data locally on EC2 instances is not suffcient since that data is not

persistent and would be lost if a server reboots. Amazon EBS volumes provide persistent storage, but a volume can only be attached

to one instance at a time a problem if your business requires hundreds of web servers to be provisioned in a few minutes. Envoy

needed to ensure their data was easily shared.

our server infrastructure, but in terms of addressing our storage requirements Gluster was

the only solution on the market that allowed us to tie the storage pieces id Michael

Taggart, chief technology officer at Envoy Media Gr -to-use solution that could integrate with AWS and

deliver the scalability and cost-effectiveness we were looking for. solution was able to deliver these

benefits

Solution Envoy deployed an integrated solution using Amazon EC2, EBS,

and S3 for backup. Gluster provides the network fle system for

the web servers and computational servers running in the EC2

instances. Media files are stored on EBS volumes which are

mounted via four Gluster storage nodes and can be accessed and

shared across as many EC2 web server instances as necessary.

The Gluster storage servers are organized in pairs, with data

replicated between each pair for high availability and the ability

to survive server failure. Additionally, data is backed up to S3.

-to-use interface and fexibility made the installation

Gluster is POSIX compliant we did not need to make any changes to our existing storage problem

with software and are not letting the hardware dictate how we operate or design

We needed an easy-to-use solution

that could integrate with AWS and

deliver the scalability and cost-

effectiveness we were looking for.

storage solution

was able to deliver these benefits

EC2 Instance Store or Ephemeral Volumes

Free with your EC2 Instance

Zero Network Overhead; local, direct attached resource.

High performance for sequential reads and writes

Temporary; goes away when your instance does

Instance Store Best Practices

App-level replication

Striping (RAID 0, 1)

Block-device-mapping

Instance Store Use Cases and Users

Stateless Web Hosts

Transcoding

Caching

HPC

DNFS

Better Together: Cross-system patterns

EBS <-> S3: Snapshots

S3 <-> SimpleDB: meta-data store

S3 <-> Local Storage: Caching

Two types of EC2 AMIs and Instances:

Instance store (“S3-backed”)Original EC2, boot from ephemeral storage

Can Start and Terminate only

All data is ephemeral (unless separate EBS volume attached)

EBS root (“boot from EBS”)Newer, boot from EBS volume

Can Start, Stop, Create Image, and Terminate

Ephemeral storage still available, but not exposed by defaultUse instance Block Device Mapping (command line and API only)

Additional EBS volumes can be attached to either type

Short Detour: S3 vs EBS AMIs

So, which should I choose?

Might be all of them!

S3: Static Online Content, Media Serving, Backups, Big Data, Centralized Resources

EBS: Applications that require POSIX, Databases, Data Warehouses

Ephemeral: Cache/Swap/Temp, Applications that require minimum latency

Test!

It costs pennies to evaluate these technologies against your use-case, and making the right choice could save you millions.

AWS Import/Export

Faster than Internet transfer and more cost effective than upgrading your connectivity

Use cases: data migration, offsite backup, direct data interchange, disaster recovery

WAN Optimization

Asperatech, Riverbed, Tsunami-UDP, etc

How do I get it to the Cloud?

Read the originals!

Storage Options in the AWS Cloud:

https://media.amazonwebservices.com/AWS_Storage_Options.pdf

Storage Use Cases in the AWS Cloud:

https://media.amazonwebservices.com/AWS_Storage_Use_Cases.pdf

Thanks!

http://aws.amazon.com