© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Paul Reed – Storage Gateway Product Manager

18 April 2017

AWS Storage Gateway

Cloud Data Migration

Direct

Connect

Snow* data

transport

family

3rd Party

Connectors

Transfer

Acceleration

Storage

GatewayKinesis Firehose

AWS Storage Platform and SolutionsThe AWS Storage Portfolio

Object

Amazon GlacierAmazon S3

Block

Amazon EBS (persistent)

Amazon EC2

Instance Store (ephemeral)

File

Amazon EFS

Hybrid storage use cases and architectures

Enabling cloud workloadsMove data to AWS storage for Big Data, cloud bursting, or migration

Tiered cloud storageEasily add AWS storage to your on-premises environment

Backup, archive, and disaster recoveryCost effective storage in AWS with local or cloud restore

Storage Gateway hybrid storage solutionsEnables using standard storage protocols to access AWS storage services

AWS Storage

Gateway

Amazon EBS

snapshots

Amazon S3

Amazon Glacier

AWS Identity and Access

Management (IAM)

AWS Key Management

Service (KMS)

AWS

CloudTrail

Amazon

CloudWatch

Files

Volumes

Tapes

Storage Gateway – Files, volumes, and tapes

File gateway NFS (v3 and v4.1) interfaceOn-premises file storage backed by Amazon S3 objects

Tape gateway iSCSI virtual tape library interfaceVirtual tape storage in Amazon S3 and Glacier with VTL management

Volume gateway iSCSI block interfaceOn-premises block storage backed by S3 with EBS snapshots

Storage Gateway – Common capabilities

Standard storage protocols integrate with on-premises applications

Local caching for low-latency access to frequently used data

Efficient data transfer with buffering and bandwidth management

Native data storage in AWS

Stateless virtual appliance for resiliency

Integrated with AWS management and security

File gatewayOn-premises file storage maintained as objects in Amazon S3

Customer Premises

File

Gateway

Data stored and retrieved from your S3 buckets

One-to-one mapping from files-to-objects

File metadata stored in object metadata

Bucket access through IAM role you own and manage

Use S3 Lifecycle Policies, versioning, or CRR to manage data

GlacierS3

Standard

S3

Standard -

Infrequent

Access

HTTPSNFS

v3 / v4.1

Application

Server

Enabling cloud workloadsMove data to AWS storage for Big Data, cloud bursting, or migration

“Storage Gateway has the promise to transform the way we move

data into the cloud. The NFS interface lets us easily integrate data

files from analytical instruments, and the transparent S3 storage

lets us easily connect our cloud-based applications and leverage the

powerful storage capabilities of S3.

With Storage Gateway, we can now unleash the full power of AWS

on our instrument data.”

S3 Bucket

Storage

Gateway

S3 Bucket

Read-only NFS client

Cross region

replication

NFS clientAWS Snowball

S3 Bucket

AWS Snowball S3 cross-region replication

Storage

Gateway

Read-only NFS client

Job

GETs/PUTs

objects

RefreshCache

Amazon

EMR

In-cloud workload

S3 Bucket

Storage

Gateway

RefreshCache RefreshCache

File Gateway – CacheRefresh

Application

Server

Volume GatewayOn-premises volume storage backed by Amazon S3 with EBS snapshots

Block storage in S3 accessed via the volume gateway

Data compressed in-transit and at-rest

Backup on-premises volumes to EBS snapshots

Create on-premises volumes from EBS snapshots

Up to 1PB of total volume storage per gateway

Amazon

EBS

snapshots

Storage Gateway

bucket in

Amazon S3

Customer Premises

Volume

Gateway

iSCSI HTTPS

Customer data center

AWS Storage

Gateway VM

Volume Gateway – Stored

Primary data stored on-premises

Asynchronous upload to AWS

Point-in-time backups stored as Amazon EBS snapshots

Up to 32 volumes, up to 16 TB each, for up to 512 TB per gateway

Amazon EBS

snapshots

Application

server

INIT

IAT

OR

TA

RG

ET

Upload

Buffer

Volume

Storage

AWS

Storage Gateway

service

Customer data center

AWS Storage

Gateway VM

Volume Gateway – Cached

Primary data stored in AWS

Frequently accessed data cached on-premises

Point-in-time backups stored as Amazon EBS snapshots

Up to 32 volumes, up to 32 TB each, for up to 1 PB per gateway

Amazon EBS

snapshots

Application

server

INIT

IAT

OR

TA

RG

ET

Upload

Buffer

Cache

Storage

Volume storage

backed by

Amazon S3

AWS

Storage Gateway

service

Tiered storage to S3, Glacier, and EBS Easily add AWS storage to your on-premises environment

“Storage Gateway is at the core of our disaster

recovery and business continuity (BCM) processes,

handling our co-lo'd OLTP and OLAP off site data

backups, as well as our in-office BCM. It works

transparently, in a lights out way, archiving off to a

separate AWS account with a simple grandfather-father-

son snapshot plan in place”.

Tape gatewayVirtual tape storage in Amazon S3 and Glacier with VTL management

Virtual tape storage in S3 and Glacier accessed via tape gateway

Data compressed in-transit and at-rest

Up to 1 PB total tape storage per gateway, unlimited archive capacity

Supports leading backup applications:

Archived Tapes

stored in

Amazon Glacier

ME

DIA

CH

AN

GE

R

TA

PE

DR

IVE

Customer Premises

Tape

Gateway

Virtual Tapes

stored in

Amazon S3

Backup

Server

HTTPSiSCSI

Backup, archive, and disaster recoveryCost effective storage in AWS with local or cloud restore

“Tapes are a headache, prone with hardware

failures, offsite storage costs, and constant

maintenance needs. Storage Gateway

provided the most cost-effective and simple

alternative. We even got disaster recovery by

using a bi-coastal data center”.

Storage Gateway – Key Benefits

Seamless integration across standard storage protocols

Low-latency access

Durability, cost, and elasticity of AWS Storage services

Efficient data transfer

Data encryption

Integrated with AWS monitoring, management, and security

Storage Gateway – Pricing

All gateway types $0.01 per GB of data written to AWS*

File Files stored and billed by S3

Volume $0.023 per GB-month of volume data stored

Snapshots stored and billed by EBS

Tape $0.023 per GB-month of tape data stored

$0.004 per GB-month of tape data archived

$0.01 per GB of data retrieved from archive

* Up to a maximum of $125/month. First 100GB per gateway free.

Under the hood

File Gateway – File system metadata

File system metadata persisted in object user-metadata, eg.

Configurable defaults for objects that don’t have this metadata• E.g. objects that were already in the bucket

Changing file metadata copies the object

x-amz-meta-file-permissions: 0666

x-amz-meta-file-user-class: 4321

x-amz-meta-file-group-class: 42

x-amz-meta-file-created: 2016-10-05T20:08:45+00:00

x-amz-meta-file-last-modified: 2016-10-05T20:08:45+00:00

How do I monitor cache performance?

CloudWatch metrics metrics for gatewayCachePercentUsed

CacheHitPercent

CachePercentDirty

High CachePercentUsed is goodOnce 100% will start to impact CacheHitPercent

May indicate the cache is too small for the working set

Writes increase CachePercentDirtyReduces size of working cache, and can reduce CacheHitPercent

May indicate data not uploading to AWS quickly enough

How does Storage Gateway transfer data to

AWS?Dirty data asynchronously uploaded to AWS

Byte-level parallel upload and download

Compression for tape and volume

Uploaded data committed to storage resource periodically

File uses multi-part PUT

Tape and volume create periodic recovery points (internal snapshots)

Rate of writes by application determine periodicity of commit

High write rates will commit more frequently

Low write rates will commit based on timer

How do I monitor data transfer?

CloudWatch metrics for gateway and per storage resource

For volumes TimeSinceLastRecoveryPoint indicates how long since the last commit

Application

serverStorage Gateway

WriteBytesReadBytes

CloudBytesUploadedCloudBytesDownloaded

Monitoring and Security extended to on-

premises gateways

IAM Integration for File Gateway rolesCloudTrail Logging of API Calls

Getting Started

AWS Storage Gateway home page:

aws.amazon.com/storagegateway

AWS Storage Gateway documentation:

aws.amazon.com/documentation/storage-gateway/

Ready to get started? AWS Storage Gateway console:

console.aws.amazon.com/storagegateway/home

Storage Gateway

Thanks!

Simple, Secure, Cost-effective Hybrid Storage in AWS