(stg311) aws storage gateway: secure, cost-effective backup & archive

24
© 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Paul Reed, AWS Storage Gateway October 2015 AWS Storage Gateway Deep Dive Simple, Secure, Cost-effective Storage in AWS STG311

Upload: amazon-web-services

Post on 22-Jan-2018

2.668 views

Category:

Technology


0 download

TRANSCRIPT

Page 1: (STG311) AWS Storage Gateway: Secure, Cost-Effective Backup & Archive

© 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Paul Reed, AWS Storage Gateway

October 2015

AWS Storage Gateway Deep Dive

Simple, Secure, Cost-effective Storage in AWS

STG311

Page 2: (STG311) AWS Storage Gateway: Secure, Cost-Effective Backup & Archive

Agenda

• AWS Storage Gateway overview

• Demo setup and activation

• Monitoring and managing

• Where to learn more

Page 3: (STG311) AWS Storage Gateway: Secure, Cost-Effective Backup & Archive

AWS Storage Gateway overview

Page 4: (STG311) AWS Storage Gateway: Secure, Cost-Effective Backup & Archive

What is AWS Storage Gateway?

Works with your existing applications

Secure and durable storage in AWS

Low-latency for frequently used data

Scalable and cost-effective on-premises storage

Service connecting an on-premises software appliance

with cloud-based storage

Page 5: (STG311) AWS Storage Gateway: Secure, Cost-Effective Backup & Archive

Common uses for AWS Storage Gateway

Backup and archive

Disaster recovery

Data migration or mirroring

Replace or expand on-premises storage

Page 6: (STG311) AWS Storage Gateway: Secure, Cost-Effective Backup & Archive

How does AWS Storage Gateway work?

Amazon EBS

snapshots

Amazon S3

Amazon

Glacier

AWS

Storage Gateway

appliance

Application

server

AWS

Storage Gateway

backend

AWS

Direct

Connect

Internet

Customer premises

Page 7: (STG311) AWS Storage Gateway: Secure, Cost-Effective Backup & Archive

Amazon EBS

snapshots

Amazon S3

Amazon

Glacier

AWS

Storage Gateway

appliance

Application

server

AWS

Storage Gateway

backend

What happens when my application writes data?

iSCSI SSL

AWS Storage Gateway

stores blocks locally, and

asynchronously

compresses and securely

uploads the changed data

Application writes data

to AWS Storage

Gateway using the

iSCSI block protocol

Backend takes

compressed data and

stores in Amazon S3,

Amazon Glacier, or

Amazon EBS, with AES-

256 encryption

321

21 3

Customer premises

Page 8: (STG311) AWS Storage Gateway: Secure, Cost-Effective Backup & Archive

Amazon EBS

snapshots

Amazon S3

Amazon

Glacier

AWS

Storage Gateway

appliance

Application

server

AWS

Storage Gateway

backend

Customer premises

What happens when my application reads data?

iSCSI SSL

Data not in local storage is

requested from backend

Application reads data

from AWS Storage

Gateway using the

iSCSI block protocol

Backend fetches

compressed data from

Amazon S3, Amazon

Glacier, or Amazon

EBS

4

31 4

AWS Storage Gateway

receives data from backend,

decompresses, stores

locally, and responds to the

application

5

1

AWS Storage Gateway

returns requested data

from local storage

2

2 35

Page 9: (STG311) AWS Storage Gateway: Secure, Cost-Effective Backup & Archive

AWS Storage Gateway configurations

iSCSI block storage

Gateway-stored volumes

iSCSI virtual tape storage

Low-latency for all your data with point-in-time

backups to AWS

Replacement for on-premises physical tape

infrastructure for backup and archive

Gateway-cached volumes

Gateway-virtual tape library (VTL)

Low-latency for frequently used data with all data

stored in AWS

Page 10: (STG311) AWS Storage Gateway: Secure, Cost-Effective Backup & Archive

Customer data center

AWS Storage

Gateway VM

Gateway-stored volumes

• Primary data stored on-premises

• Asynchronous backup to AWS

• Point-in-time backups stored as Amazon EBS snapshots

• Up to 12 volumes, up to 16 TB each, for up to 192 TB per gateway

Amazon EBS

snapshots

Application

server

INIT

IAT

OR

TA

RG

ET

Upload

Buffer

Volume

Storage

AWS

Storage Gateway

service

Page 11: (STG311) AWS Storage Gateway: Secure, Cost-Effective Backup & Archive

Customer data center

AWS Storage

Gateway VM

Gateway-cached volumes

• Primary data stored in AWS

• Frequently accessed data cached on-premises

• Point-in-time backups stored as Amazon EBS snapshots

• Up to 32 volumes, up to 32 TB each, for up to 1 PB per gateway

Amazon EBS

snapshots

Application

server

INIT

IAT

OR

TA

RG

ET

Upload

Buffer

Cache

Storage

Volume storage

backed by

Amazon S3

AWS

Storage Gateway

service

Page 12: (STG311) AWS Storage Gateway: Secure, Cost-Effective Backup & Archive

Gateway-virtual tape library (VTL)

• Virtual tapes stored in AWS

• Frequently accessed data cached on-premises

• Up to 1,500 tapes, up to 2.5 TB each, for up to 150 TB per gateway-VTL

• Unlimited number of tapes in virtual tape shelf (VTS)

Customer data center

VTS storage

backed by

Amazon Glacier

AWS Storage

Gateway VM

Backup

Server

INIT

IAT

OR

AWS

Storage Gateway

service

ME

DIA

CH

AN

GE

R

Upload

Buffer

Cache

Storage

Gateway-VTL

storage backed

by Amazon S3

VTS

TA

PE

DR

IVE

Page 13: (STG311) AWS Storage Gateway: Secure, Cost-Effective Backup & Archive

Demo setup and activation

Page 14: (STG311) AWS Storage Gateway: Secure, Cost-Effective Backup & Archive

Monitoring and managing

AWS Storage Gateway

Page 15: (STG311) AWS Storage Gateway: Secure, Cost-Effective Backup & Archive

Using system resource check on your VM

• Checks virtual system resources available to the gateway

• AWS Storage Gateway won’t start if minimum resources are not

provisioned

Page 16: (STG311) AWS Storage Gateway: Secure, Cost-Effective Backup & Archive

Using Amazon CloudWatch to monitor performance

• Monitoring and set up alarms for your gateway

• 27 metrics reported for gateways, volumes, and tapes

• Available through AWS console, API, AWS CLI, and SDK

Page 17: (STG311) AWS Storage Gateway: Secure, Cost-Effective Backup & Archive

Monitoring write performance with Amazon CloudWatch

• Changed data stored in both cache and upload buffer

• Size local storage to match application and network throughput

• Writes are throttled as upload buffer gets close to full

• Overwrites collapsed to expedite upload of latest changed data

WriteBytesWriteTime

TimeSinceLastRecoveryPoint

UploadBufferPercentUsedCachePercentDirty

Application

server

AWS Storage Gateway

Learn more: http://docs.aws.amazon.com/storagegateway/latest/userguide/GatewayCachedLocalStorage.html

Page 18: (STG311) AWS Storage Gateway: Secure, Cost-Effective Backup & Archive

Monitoring read performance with Amazon CloudWatch

• Cache should hold application working set of storage

• Strive for high cache hit rate and high cache utilization

• Writes are stored in cache so can impact read performance

ReadBytesReadTime

CloudBytesDownloadedCloudDownloadLatency

CacheHitPercentCachePercentUsedCachePercentDirty

Application

server

AWS Storage Gateway

Learn more: http://docs.aws.amazon.com/storagegateway/latest/userguide/GatewayCachedLocalStorage.html

Page 19: (STG311) AWS Storage Gateway: Secure, Cost-Effective Backup & Archive

Using AWS CloudTrail to monitor events

AWS Storage Gateway API calls can be logged with AWS CloudTrail

Page 20: (STG311) AWS Storage Gateway: Secure, Cost-Effective Backup & Archive

Getting notifications for specific events

AWS CloudTrail saves

logs to an Amazon S3

bucket

Amazon S3 bucket notification

configuration publishes event

to AWS Lambda which invokes

function

AWS CloudTrail

Amazon S3

Function

AWS Lambda

Lambda function reads log

and sends Amazon SNS

notification of specific AWS

Storage Gateway events

1

2

3

3

Amazon SNS

TopicLog Bucket

1 2

Learn more: http://docs.aws.amazon.com/lambda/latest/dg/wt-cloudtrail-events-adminuser.html

3

Page 21: (STG311) AWS Storage Gateway: Secure, Cost-Effective Backup & Archive

Where to learn more

Page 22: (STG311) AWS Storage Gateway: Secure, Cost-Effective Backup & Archive

Resources

Documentation

http://aws.amazon.com/storagegateway/

Developer Forum

http://forums.aws.amazon.com/

Developer Support

https://aws.amazon.com/premiumsupport/

Try it!

AWS Free Tier

https://aws.amazon.com/free/

Page 23: (STG311) AWS Storage Gateway: Secure, Cost-Effective Backup & Archive

Remember to complete

your evaluations!

Page 24: (STG311) AWS Storage Gateway: Secure, Cost-Effective Backup & Archive

Thank you!