© 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Paul Reed, AWS Storage Gateway
October 2015
AWS Storage Gateway Deep Dive
Simple, Secure, Cost-effective Storage in AWS
STG311
Agenda
• AWS Storage Gateway overview
• Demo setup and activation
• Monitoring and managing
• Where to learn more
AWS Storage Gateway overview
What is AWS Storage Gateway?
Works with your existing applications
Secure and durable storage in AWS
Low-latency for frequently used data
Scalable and cost-effective on-premises storage
Service connecting an on-premises software appliance
with cloud-based storage
Common uses for AWS Storage Gateway
Backup and archive
Disaster recovery
Data migration or mirroring
Replace or expand on-premises storage
How does AWS Storage Gateway work?
Amazon EBS
snapshots
Amazon S3
Amazon
Glacier
AWS
Storage Gateway
appliance
Application
server
AWS
Storage Gateway
backend
AWS
Direct
Connect
Internet
Customer premises
Amazon EBS
snapshots
Amazon S3
Amazon
Glacier
AWS
Storage Gateway
appliance
Application
server
AWS
Storage Gateway
backend
What happens when my application writes data?
iSCSI SSL
AWS Storage Gateway
stores blocks locally, and
asynchronously
compresses and securely
uploads the changed data
Application writes data
to AWS Storage
Gateway using the
iSCSI block protocol
Backend takes
compressed data and
stores in Amazon S3,
Amazon Glacier, or
Amazon EBS, with AES-
256 encryption
321
21 3
Customer premises
Amazon EBS
snapshots
Amazon S3
Amazon
Glacier
AWS
Storage Gateway
appliance
Application
server
AWS
Storage Gateway
backend
Customer premises
What happens when my application reads data?
iSCSI SSL
Data not in local storage is
requested from backend
Application reads data
from AWS Storage
Gateway using the
iSCSI block protocol
Backend fetches
compressed data from
Amazon S3, Amazon
Glacier, or Amazon
EBS
4
31 4
AWS Storage Gateway
receives data from backend,
decompresses, stores
locally, and responds to the
application
5
1
AWS Storage Gateway
returns requested data
from local storage
2
2 35
AWS Storage Gateway configurations
iSCSI block storage
Gateway-stored volumes
iSCSI virtual tape storage
Low-latency for all your data with point-in-time
backups to AWS
Replacement for on-premises physical tape
infrastructure for backup and archive
Gateway-cached volumes
Gateway-virtual tape library (VTL)
Low-latency for frequently used data with all data
stored in AWS
Customer data center
AWS Storage
Gateway VM
Gateway-stored volumes
• Primary data stored on-premises
• Asynchronous backup to AWS
• Point-in-time backups stored as Amazon EBS snapshots
• Up to 12 volumes, up to 16 TB each, for up to 192 TB per gateway
Amazon EBS
snapshots
Application
server
INIT
IAT
OR
TA
RG
ET
Upload
Buffer
Volume
Storage
AWS
Storage Gateway
service
Customer data center
AWS Storage
Gateway VM
Gateway-cached volumes
• Primary data stored in AWS
• Frequently accessed data cached on-premises
• Point-in-time backups stored as Amazon EBS snapshots
• Up to 32 volumes, up to 32 TB each, for up to 1 PB per gateway
Amazon EBS
snapshots
Application
server
INIT
IAT
OR
TA
RG
ET
Upload
Buffer
Cache
Storage
Volume storage
backed by
Amazon S3
AWS
Storage Gateway
service
Gateway-virtual tape library (VTL)
• Virtual tapes stored in AWS
• Frequently accessed data cached on-premises
• Up to 1,500 tapes, up to 2.5 TB each, for up to 150 TB per gateway-VTL
• Unlimited number of tapes in virtual tape shelf (VTS)
Customer data center
VTS storage
backed by
Amazon Glacier
AWS Storage
Gateway VM
Backup
Server
INIT
IAT
OR
AWS
Storage Gateway
service
ME
DIA
CH
AN
GE
R
Upload
Buffer
Cache
Storage
Gateway-VTL
storage backed
by Amazon S3
VTS
TA
PE
DR
IVE
Demo setup and activation
Monitoring and managing
AWS Storage Gateway
Using system resource check on your VM
• Checks virtual system resources available to the gateway
• AWS Storage Gateway won’t start if minimum resources are not
provisioned
Using Amazon CloudWatch to monitor performance
• Monitoring and set up alarms for your gateway
• 27 metrics reported for gateways, volumes, and tapes
• Available through AWS console, API, AWS CLI, and SDK
Monitoring write performance with Amazon CloudWatch
• Changed data stored in both cache and upload buffer
• Size local storage to match application and network throughput
• Writes are throttled as upload buffer gets close to full
• Overwrites collapsed to expedite upload of latest changed data
WriteBytesWriteTime
TimeSinceLastRecoveryPoint
UploadBufferPercentUsedCachePercentDirty
Application
server
AWS Storage Gateway
Learn more: http://docs.aws.amazon.com/storagegateway/latest/userguide/GatewayCachedLocalStorage.html
Monitoring read performance with Amazon CloudWatch
• Cache should hold application working set of storage
• Strive for high cache hit rate and high cache utilization
• Writes are stored in cache so can impact read performance
ReadBytesReadTime
CloudBytesDownloadedCloudDownloadLatency
CacheHitPercentCachePercentUsedCachePercentDirty
Application
server
AWS Storage Gateway
Learn more: http://docs.aws.amazon.com/storagegateway/latest/userguide/GatewayCachedLocalStorage.html
Using AWS CloudTrail to monitor events
AWS Storage Gateway API calls can be logged with AWS CloudTrail
Getting notifications for specific events
AWS CloudTrail saves
logs to an Amazon S3
bucket
Amazon S3 bucket notification
configuration publishes event
to AWS Lambda which invokes
function
AWS CloudTrail
Amazon S3
Function
AWS Lambda
Lambda function reads log
and sends Amazon SNS
notification of specific AWS
Storage Gateway events
1
2
3
3
Amazon SNS
TopicLog Bucket
1 2
Learn more: http://docs.aws.amazon.com/lambda/latest/dg/wt-cloudtrail-events-adminuser.html
3
Where to learn more
Resources
Documentation
http://aws.amazon.com/storagegateway/
Developer Forum
http://forums.aws.amazon.com/
Developer Support
https://aws.amazon.com/premiumsupport/
Try it!
AWS Free Tier
https://aws.amazon.com/free/
Remember to complete
your evaluations!
Thank you!