deep dive on s3 storage management covering new feature announcements - december 2016 monthly...
TRANSCRIPT
![Page 1: Deep Dive on S3 Storage Management Covering New Feature Announcements - December 2016 Monthly Webinar Series](https://reader036.vdocuments.net/reader036/viewer/2022081513/586e8bb11a28aba0038b8129/html5/thumbnails/1.jpg)
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Sr. Product Manager, AWS
Deep Dive on S3 Storage ManagementOmair Gillani
![Page 2: Deep Dive on S3 Storage Management Covering New Feature Announcements - December 2016 Monthly Webinar Series](https://reader036.vdocuments.net/reader036/viewer/2022081513/586e8bb11a28aba0038b8129/html5/thumbnails/2.jpg)
Batches and Streams
AWS Direct Connect
AWS Snowball, Snowball Edge,
Snowmobile
3rd Party Connectors
Transfer Acceleration
AWS Storage Gateway
Amazon Kinesis Firehose
File BlockObject
Amazon GlacierAmazon S3 Amazon EC2 Instance Store
(ephemeral)
Amazon EFS Amazon EBS (persistent)
![Page 3: Deep Dive on S3 Storage Management Covering New Feature Announcements - December 2016 Monthly Webinar Series](https://reader036.vdocuments.net/reader036/viewer/2022081513/586e8bb11a28aba0038b8129/html5/thumbnails/3.jpg)
AWS Direct Connect AWS Snowball ISV Connectors
Amazon Kinesis Firehose
S3 Transfer Acceleration
AWS Storage Gateway
Data transfer into Amazon S3
AWS Snowmobile
AWS Snowball Edge
![Page 4: Deep Dive on S3 Storage Management Covering New Feature Announcements - December 2016 Monthly Webinar Series](https://reader036.vdocuments.net/reader036/viewer/2022081513/586e8bb11a28aba0038b8129/html5/thumbnails/4.jpg)
Storage Gateway hybrid storage solutionsEnables using standard storage protocols to access AWS storage services
Customer Premises
StorageGateway
Amazon EBS snapshots
Amazon S3Amazon Glacier
AWS Identity and Access Management (IAM)
AWS Key Management Service (KMS)
AWS CloudTrail
Amazon CloudWatch
Enterprise storage
Devices
Applicationservers
![Page 5: Deep Dive on S3 Storage Management Covering New Feature Announcements - December 2016 Monthly Webinar Series](https://reader036.vdocuments.net/reader036/viewer/2022081513/586e8bb11a28aba0038b8129/html5/thumbnails/5.jpg)
What’s new? File gatewayFile interface to objects in your Amazon S3 buckets
Virtual on-premises file server
Store and retrieve Amazon S3 objects through standard file storage protocols (NFS v3 and v4.1)
Local cache for low-latency access
Direct access to objects with native S3 capabilities, such as versioning, lifecycle management, and cross region replication
Launched
@ re:Invent!
![Page 6: Deep Dive on S3 Storage Management Covering New Feature Announcements - December 2016 Monthly Webinar Series](https://reader036.vdocuments.net/reader036/viewer/2022081513/586e8bb11a28aba0038b8129/html5/thumbnails/6.jpg)
What’s new? We’re reducing pricesEffective December 1, 2016
Storage prices reducedVolume and virtual tape storage by up to 28% Archived virtual tape storage by up to 43%
Flat per GB price replaces monthly gateway fee$0.01 per GB written to AWS storage by your gateway*
Simplified virtual tape retrieval pricing$0.01 per GB retrieved replaces peak-rate fee
* Up to $125/month per gateway
Launched
@ re:Invent!
![Page 7: Deep Dive on S3 Storage Management Covering New Feature Announcements - December 2016 Monthly Webinar Series](https://reader036.vdocuments.net/reader036/viewer/2022081513/586e8bb11a28aba0038b8129/html5/thumbnails/7.jpg)
Cross-Region ReplicationLifecycle Policy S3 Object TagsEvent Notifications
Amazon S3 CloudWatch Metrics
S3 Inventory Audit with AWS CloudTrail S3 Data Events
S3 Analytics
Standard Standard - Infrequent Access Amazon Glacier
Storage Management for S3
![Page 8: Deep Dive on S3 Storage Management Covering New Feature Announcements - December 2016 Monthly Webinar Series](https://reader036.vdocuments.net/reader036/viewer/2022081513/586e8bb11a28aba0038b8129/html5/thumbnails/8.jpg)
Understand your storage usage
S3 InventoryAnalyze Logs with Amazon EMR
S3 Analytics
![Page 9: Deep Dive on S3 Storage Management Covering New Feature Announcements - December 2016 Monthly Webinar Series](https://reader036.vdocuments.net/reader036/viewer/2022081513/586e8bb11a28aba0038b8129/html5/thumbnails/9.jpg)
S3 Inventory
Save time Daily or Weekly delivery Delivery to S3 bucketCSV File Output
Trigger business workflows and applications such as secondary index, garbage collection, data auditing, and offline analytics
Half the price of LIST API at $0.0025 per million objects listed
![Page 10: Deep Dive on S3 Storage Management Covering New Feature Announcements - December 2016 Monthly Webinar Series](https://reader036.vdocuments.net/reader036/viewer/2022081513/586e8bb11a28aba0038b8129/html5/thumbnails/10.jpg)
S3 Inventory
More information about your objects than provided by LIST API such as replication
status, multipart upload flag, and delete marker
Name Value Type DescriptionBucket String Bucket name. UTF-8 encoded.Key String Object key name. UTF-8 encoded.Version Id String Version Id of the objectIs Latest Boolean true if object is the latest version (current version) of a versioned object, otherwise falseDelete Marker Boolean true if object is a delete marker of a versioned object, otherwise falseSize Long Object size in bytesLast Modified String Last modified timestamp. Format in ISO: YYYY-MM-DDTHH:mm:ss.SSSZETag String eTag in HEX encoded format
StorageClass String Valid values: STANDARD, REDUCED_REDUNDANCY, GLACIER, STANDARD_IA. UTF-8 encoded.
Multipart Uploaded Boolean true if object is uploaded by using multipart, otherwise falseReplication Status String Valid values: REPLICA, COMPLETED, PENDING, FAILED. UTF-8 encoded.
![Page 11: Deep Dive on S3 Storage Management Covering New Feature Announcements - December 2016 Monthly Webinar Series](https://reader036.vdocuments.net/reader036/viewer/2022081513/586e8bb11a28aba0038b8129/html5/thumbnails/11.jpg)
![Page 12: Deep Dive on S3 Storage Management Covering New Feature Announcements - December 2016 Monthly Webinar Series](https://reader036.vdocuments.net/reader036/viewer/2022081513/586e8bb11a28aba0038b8129/html5/thumbnails/12.jpg)
S3 Inventory
Setup notification when S3 Inventory is complete
/Data /<InventoryFile>.gz /<InventoryFile>.gz …
/<DayofReport> /manifest.json /manifest.checksum …
AWS Lambda
Amazon SQS
Amazon SNS
![Page 13: Deep Dive on S3 Storage Management Covering New Feature Announcements - December 2016 Monthly Webinar Series](https://reader036.vdocuments.net/reader036/viewer/2022081513/586e8bb11a28aba0038b8129/html5/thumbnails/13.jpg)
Eventually consistent rolling snapshot
S3 Inventory
New objects may not be listed Recently deleted objects may still be included
O1
O2
O3
O1
O2
O3
O1
O2
O1
O2
O3NEW
Validate before you act!Use HEAD OBJECT or GET OBJECT
![Page 14: Deep Dive on S3 Storage Management Covering New Feature Announcements - December 2016 Monthly Webinar Series](https://reader036.vdocuments.net/reader036/viewer/2022081513/586e8bb11a28aba0038b8129/html5/thumbnails/14.jpg)
S3 Analytics – Storage Class Analysis
Analyze buckets, prefixes or tags
$0.10 per million objects analyzed per month
Daily Storage Class Analysis & Lifecycle candidates
Data-driven storage management for S3
Export Analysis data to your S3 bucket
![Page 15: Deep Dive on S3 Storage Management Covering New Feature Announcements - December 2016 Monthly Webinar Series](https://reader036.vdocuments.net/reader036/viewer/2022081513/586e8bb11a28aba0038b8129/html5/thumbnails/15.jpg)
S3 Analytics – Storage Class Analysis
![Page 16: Deep Dive on S3 Storage Management Covering New Feature Announcements - December 2016 Monthly Webinar Series](https://reader036.vdocuments.net/reader036/viewer/2022081513/586e8bb11a28aba0038b8129/html5/thumbnails/16.jpg)
S3 Analytics – Storage Class Analysis
![Page 17: Deep Dive on S3 Storage Management Covering New Feature Announcements - December 2016 Monthly Webinar Series](https://reader036.vdocuments.net/reader036/viewer/2022081513/586e8bb11a28aba0038b8129/html5/thumbnails/17.jpg)
S3 Analytics – Storage Class Analysis
Export to use BI tool of your choice
![Page 18: Deep Dive on S3 Storage Management Covering New Feature Announcements - December 2016 Monthly Webinar Series](https://reader036.vdocuments.net/reader036/viewer/2022081513/586e8bb11a28aba0038b8129/html5/thumbnails/18.jpg)
S3 Analytics – Storage Class Analysis
Daily Export Dataset
![Page 19: Deep Dive on S3 Storage Management Covering New Feature Announcements - December 2016 Monthly Webinar Series](https://reader036.vdocuments.net/reader036/viewer/2022081513/586e8bb11a28aba0038b8129/html5/thumbnails/19.jpg)
Simple to configure S3 Analytics
S3 Management Console PUT Bucket AnalyticsMultiple Policy
Documents
<AnalyticsConfiguration> <Id>...</Id> <Filter> ... </Filter> <StorageClassAnalysis> <DataExport> ... </DataExport> </StorageClassAnalysis> </AnalyticsConfiguration>
![Page 20: Deep Dive on S3 Storage Management Covering New Feature Announcements - December 2016 Monthly Webinar Series](https://reader036.vdocuments.net/reader036/viewer/2022081513/586e8bb11a28aba0038b8129/html5/thumbnails/20.jpg)
Monitor your storage
Monitor and Alert with CloudWatch
Audit your storage with CloudTrail Data Events
Server Access Logs
![Page 21: Deep Dive on S3 Storage Management Covering New Feature Announcements - December 2016 Monthly Webinar Series](https://reader036.vdocuments.net/reader036/viewer/2022081513/586e8bb11a28aba0038b8129/html5/thumbnails/21.jpg)
CloudWatch metrics for S3
Operational & performance monitoring Generate metrics for data of your choice
Entire bucket, Prefixes, and Tags Up to 1,000 object groups
1-minute CloudWatch metrics Alert and alarm on metrics
![Page 22: Deep Dive on S3 Storage Management Covering New Feature Announcements - December 2016 Monthly Webinar Series](https://reader036.vdocuments.net/reader036/viewer/2022081513/586e8bb11a28aba0038b8129/html5/thumbnails/22.jpg)
CloudWatch metrics for S3
Metric Name Metric value
AllRequests Count
PutRequests Count
PostRequests Count
GetRequests Count
ListRequests Count
DeleteRequests Count
HeadRequests Count
Metric Name Metric value
BytesDownloaded MB
BytesUploaded MB
4xxErrors Count
5xxErrors Count
FirstByteLatency ms
TotalRequestLatency ms
$0.50 per metric per month
![Page 23: Deep Dive on S3 Storage Management Covering New Feature Announcements - December 2016 Monthly Webinar Series](https://reader036.vdocuments.net/reader036/viewer/2022081513/586e8bb11a28aba0038b8129/html5/thumbnails/23.jpg)
S3 Data Events in CloudTrail
Perform security analysis, meet your IT auditing and compliance needs, and take immediate action on object-level activity to immediately improve security posture
Pricing: $1 per million data events recorded and storage charges apply
Log object level operations
Changes to bucket configurations
SNS notification for log delivery
![Page 24: Deep Dive on S3 Storage Management Covering New Feature Announcements - December 2016 Monthly Webinar Series](https://reader036.vdocuments.net/reader036/viewer/2022081513/586e8bb11a28aba0038b8129/html5/thumbnails/24.jpg)
Manage your data
Cross-Region Replication
Lifecycle Policies Event Notifications
S3 Object Tags
![Page 25: Deep Dive on S3 Storage Management Covering New Feature Announcements - December 2016 Monthly Webinar Series](https://reader036.vdocuments.net/reader036/viewer/2022081513/586e8bb11a28aba0038b8129/html5/thumbnails/25.jpg)
Manage your data
S3 Object Tags
Easily manage and control access for Amazon S3 objects Classify your data
Tag your objects with key-value pairs
Write policies once based on the type of data
AnalyzeLifecycle PolicyAccess Control
![Page 26: Deep Dive on S3 Storage Management Covering New Feature Announcements - December 2016 Monthly Webinar Series](https://reader036.vdocuments.net/reader036/viewer/2022081513/586e8bb11a28aba0038b8129/html5/thumbnails/26.jpg)
Deep dive on tags Tags are key-value pairs Maximum 10 tags per object Maximum key length—127 Unicode characters Maximum value length—255 Unicode characters Tag keys and values are case-sensitive.
2 ways to put tags via API Put objects with tag parameter, or add tag API after object is created
Simple pricing $0.01 per 10,000 tags per month
![Page 27: Deep Dive on S3 Storage Management Covering New Feature Announcements - December 2016 Monthly Webinar Series](https://reader036.vdocuments.net/reader036/viewer/2022081513/586e8bb11a28aba0038b8129/html5/thumbnails/27.jpg)
Lifecycle policies based on tags<LifecycleConfiguration> <Rule> <ID>sample-rule</ID> <Filter> <And> <Prefix>documents/</Prefix> <Tag> <Key>Project</Key> <Value>Delta</Value> </Tag> <Tag> <Key>Data type</Key> <Value>HPI</Value> </Tag> </And> </Filter> <Status>Enabled</Status> <Transition> <Days>365</Days> <StorageClass>GLACIER</StorageClass> </Transition> <Expiration> <Days>3650</Days> </Expiration> </Rule></LifecycleConfiguration>
Transition or expire storage using tags Simplify S3 lifecycle policies Filter with prefix, tag, or both
![Page 28: Deep Dive on S3 Storage Management Covering New Feature Announcements - December 2016 Monthly Webinar Series](https://reader036.vdocuments.net/reader036/viewer/2022081513/586e8bb11a28aba0038b8129/html5/thumbnails/28.jpg)
Thank you!