cloud data migration strategies - aws may 2016 webinar series
TRANSCRIPT
© 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS Business Development
May 2016
Cloud Data Migration: Eight Strategies for Getting Data into AWS
Storage is the Gravity for Cloud Applications
Amazon EFS
File
Amazon EBS Amazon EC2Instance Store
Block
Amazon S3 Amazon Glacier
Object
Data Transfer
AWS Direct Connect
ISV Connectors
Amazon Kinesis
Firehose
Storage Gateway
S3 Transfer Acceleration
AWS Storage is a Platform
AWS Snowball
AmazonCloudFront
Internet/VPN
Internet / VPN Ingest
What is Internet/VPN…
Globally Available
Default method of ingesting content into Amazon S3
Simple standards based (HTTP) connection
Use your existing internet connection
Available within a VPC for VPN connectivity
Acceleration via Multipart Upload
Data Transfer Into AWS is free
VPN Connections using VPC Virtual Private Gateway• $0.05 per VPN Connection-hour• $0.048 per VPN Connection-hour for connections to the Tokyo region
How does Internet / VPN ingest work?Accelerate Data Transfer using
Multipart UploadIngest Data Directly Into S3 Buckets with
existing internet connectivity
S3 BucketAWS Region
and
Via Management Console or API
customer gateway
endpoints
VPN connection
Internet Internet via VPN + VPC
Amazon S3 Transfer Acceleration
What is Amazon S3 Transfer Acceleration…
Network and Protocol Based Data Transfer Service
Acceleration of Data Ingress / Egress with S3 Buckets
Typically 50% to 400% faster
Feature of S3 Enabled at the Bucket Level
Available at All S3 Regions Worldwide
No Client / Server Software Required
No Code Changes to Your Application
No Firewall Exceptions
Simple Pricing Model
Ingest & Egress with S3 transfer acceleration
S3 BucketAWS EdgeLocation
Uploader
OptimizedThroughput!
Uses AWS 55 global edge locations
AWS determines best edge location
Data transfer optimized between edge and customer, and edge and S3
Data is not stored on the edge cache
Amazon Route 53
Resolve b1.s3-accelerate.amazonaws.com
HTTPS PUT/POST upload_files.zip
HTTP/S PUT/POST “upload_files.zip”
Service traffic flowClient to S3 Bucket example
S3 Bucket
b1.s3-accelerate.amazonaws.com
EC2 Proxy
AWS Region
AWS Edge Location
Customer Client
1
2
3
4
Data is not cached on the AWS Edge location
Fully Managed File Transfer Acceleration using all AWS Edge
Locations
Using the Service is as easy as 1, 2, 3…
Update Application to Point to new S3 URL• Update“bucket.s3.amazonaws.com” to
“<bucket-name>.s3-accelerate.amazonaws.com”• Original bucket location and contents are the same, only
namespace changes
Or Use Permissions via API s3:PutAccelerateConfiguration
Enable the Service in the Management Console
Start Uploading Data to Amazon S3
1
2
3
How fast is S3 transfer acceleration?
Rio De Janeiro Warsaw New York Atlanta Madrid Virginia Melbourne Paris Los Angeles Seattle Tokyo Singapore
500 GB upload from these edge locations to a bucket in Singapore
Tim
e [h
rs]
Public InternetS3 Transfer Acceleration
How much will it help me?
Speed Checker Demo
Pricing*
Dimension Price / GB
Data Transfer In from Internet** $0.04 (Edge location in US, EU, JP)$0.08 (Edge location in rest of the world)
Data Transfer Out to Internet $0.04
Data Transfer Out to Another AWS Region $0.04
Amazon S3 Charges Standard data transfer charges apply
*Plus standard Amazon S3 data transfer charges apply**Accelerated performance or there is no bandwidth charge
Amazon CloudFront
Global Content Delivery Network
55 Edge Locations Worldwide
Supports Ingest via PUT/POST methods
Works with S3 Multi-part upload
Supports SSL SNI and TLS connections
Integrated with ACM and AWS WAF for additional security
Proxy ingest to S3, EC2 and even your own origins
Tiered and Custom Pricing Models
What is Amazon CloudFront…
Using CloudFront to Ingest Data into AWS
AWS Region
Customer Client
HTTP/S PUT/POST “upload_files.zip”
Amazon EC2
S3 Bucket
ELB
Custom Origin
CloudFront Edge Location
Ingest content into S3, EC2, ELB or your own custom origin with Amazon CloudFront
Use cache behaviors to direct to the correct origin based on PATH pattern matching
Restrict Access via Geo Restriction or AWS WAF Web ACL
Amazon CloudFront Pricing
Data Transfer out of Amazon CloudFront to your origin server billed at the “Regional Data Transfer Out to Origin” rates listed in the Regional Data Transfer Out to Origin (per GB) table.
Data Transfer out of Amazon CloudFront to Internet will be charged at rates listed in “Regional Data Transfer Out to Internet (per GB)” table.
Amazon CloudFront offers additional pricing options via a CloudFront Reserved Capacity (CFRC) contract. Contact sales for additional details and pricing.
AWS Direct Connect
Dedicated, 1 or 10 GE private pipes into AWS
Create private (VPC) or public virtual interfaces to AWS
Reduced data-out rates (data-in still free)
Consistent network performance
At least 1 location to each AWS region
Option for redundant connections
Uses BGP to exchange routing information over a VLAN
What is AWS Direct Connect…
Physical Connection
• Cross Connect at the location
• Single Mode Fiber - 1000Base-LX or 10GBASE-LR
• Potential onward Delivery via Direct Connect Partner
• Customer Router
At the Direct Connect Location
CORP
AWS Direct
Connect Routers
Customer
Router
Colocation
DX Location
CustomerNetwork`
AWS BackboneNetwork
Cross Connect
Customer Router
Customers Network
Demarcation
Dedicated Port via Direct Connect Partner
AWS Direct
Connect Routers
Colocation
DX Location
Partner Network
AWS BackboneNetwork
Cross Connect
Customer Router
PartnerNetwork
AccessCircuit
Demarcation
PartnerEquipment
CORP
Direct Connect - LocationsAWS Region AWS Direct Connect LocationAsia Pacific (Singapore) Equinix SG2, GPX, MumbaiAsia Pacific (Seoul) KINX, SeoulAsia Pacific (Sydney) Equinix SY3, Global SwitchAsia Pacific (Tokyo) Equinix OS1, Equinix TY2China (Beijing) Sinnet JiuXianqiao IDC, CIDS Jiachuang IDCEU (Frankfurt) Equinix FR5, Interxion FrankfurtEU (Ireland) TelecityGroup, London Docklands’, Eircom Clonshaugh
Equinix LD4 - LD6, LondonSouth America (Sao Paulo) Terremark NAP do Brasil, TivitUS East (Virginia) CoreSite NY1 & NY2, Equinix DC1 - DC6 & DC10US West (Northern California)
CoreSite One Wilshire & 900 North Alameda, CA, Equinix SV1 & SV5
US West (Oregon) Equinix SE2 & SE3, Switch SUPERNAP, Las VegasAWS GovCloud (US) Equinix SV1 & SV5
Amazon Kinesis Firehose
Amazon Kinesis PlatformAmazon Kinesis streaming data on the AWS cloud• Amazon Kinesis Streams• Amazon Kinesis Firehose • Amazon Kinesis Analytics
Amazon Kinesis FirehoseLoad massive volumes of streaming data into Amazon S3 and Amazon Redshift
Zero administration: Capture and deliver streaming data into S3, Redshift, and other destinations without writing an application or managing infrastructure.
Direct-to-data store integration: Batch, compress, and encrypt streaming data for delivery into data destinations in as little as 60 secs using simple configurations.
Seamless elasticity: Seamlessly scales to match data throughput w/o intervention
Capture and submit streaming data to Firehose
Firehose loads streaming data continuously into S3 and Redshift
Analyze streaming data using your favorite BI tools
Vertical/Use Case Accelerated Ingest-Load to final destination for Analytics
Ad Tech/ Marketing Analytics
Advertising data aggregation
Consumer Online/Gaming
Online customer engagement data aggregation
Financial Services Market/ Financial Transaction order data collection
IoT / Sensor Data Fitness device , vehicle Sensor, telemetry data ingestion
Amazon Kinesis Firehose Use Cases
AWS Storage Gateway
What is AWS Storage Gateway?
Works with your existing applications
Secure and durable storage in AWS
Low-latency for frequently used data
Scalable and cost-effective on-premises storage - $125 per gateway per month + S3/Glacier storage fees
Service connecting an on-premises software appliance with cloud-based storage
Common uses for AWS Storage Gateway
Backup and archive
Disaster recovery
Data migration
How does AWS Storage Gateway work?
Amazon EBS snapshots
Amazon S3Amazon Glacier
AWSStorage Gateway
appliance
Applicationserver
AWSStorage Gateway
backend
Customer premises
S3 Transfer
Acceleration
AWSDirect
Connect
Internet
AWS Storage Gateway configurations
iSCSI block storage
Gateway-stored volumes
iSCSI virtual tape storage
Low-latency for all your data with point-in-time backups to AWS
Replacement for on-premises physical tape infrastructure for backup and archive
Gateway-cached volumes
Gateway-virtual tape library (VTL)
Low-latency for frequently used data with all data stored in AWS
Gateway-virtual tape library (VTL)• Replace or augment your aging tape infrastructure with durable object
storage• Virtual tapes stored in AWS. Frequently accessed data cached on-premises• Up to 1,500 tapes, up to 2.5 TB each, for up to 150 TB per gateway-VTL• Unlimited number of tapes in virtual tape shelf (VTS)
Customer data center
VTS storage backed by Amazon Glacier
AWS Storage Gateway VM
BackupServer IN
ITIA
TOR
AWSStorage Gateway
service
MED
IA
CHAN
GER
UploadBuffer
CacheStorage
Gateway-VTLstorage backedby Amazon S3
VTS
TAPE
DR
IVE
AWS Snowball
What is AWS Snowball? Petabyte-scale data transport
E-ink shipping label
Ruggedized case“8.5G impact”
All data encrypted end-to-end
Rain- and dust-resistant
Tamper-resistant case and
electronics
80 TB10 GE network
How it works
• Less than 1 day to transfer 200TB via 3x10G connections with 3 Snowballs, less than 1 week including shipping
• Number of days to transfer 200TB via the Internet at typical utilizations
How fast is Snowball?
Internet Connection SpeedUtilization 1Gbps 500Mbps 300Mbps 150Mbps
25% 71 141 236 471
50% 36 71 118 236
75% 24 47 225 157
Use cases: AWS Snowball
Cloud Migration
Disaster Recovery
Data CenterDecommission
ContentDistribution
Pricing
Dimension Price
Usage Charge per Job $200.00 (50 TB)$250.00 (80 TB)
Extra Day Charge (First 10 days* are free) $15.00
Data Transfer In $0.00/GB
Data Transfer Out $0.03/GB
Shipping** Varies
Amazon S3 Charges Standard storage and request fees apply
* Starts one day after the appliance is delivered to you. The first day the appliance is received at your site and the last day the appliance is shipped out are also free and not included in the 10-day free usage time. ** Shipping charges are based on your shipment destination and the shipping option (e.g., overnight, 2-day) you choose.
AWS Technology Partnerships
Amazon Storage Partner Ecosystem
Gateway/NAS DataManagement Sync and ShareBackup/DR Content and
Acceleration
Archive
File System
Example of Data Transfer with Partner Solution: Attunity Cloudbeam for AWS
S3 EMR
Hourly Model, BYOL, and Free Trial Availablehttp://www.attunity.com/cloud-solutions-for-amazon-web-services
Backup to AWS Approaches
Amazon S3
Amazon GlacierAWS
DirectConnect
InternetAmazon S3-IA
Applicationservers
Cloud Gateway
Local disk
MediaServer
Cloud Gateway
HTTPS/API
Applicationservers
Backup SW cloud connector
Local diskMedia
Server with cloud
connector
HTTPS/API
CommVault Ties Together On Premise and Cloud Data StrategiesCommvault Orchestrates the Enterprise
• Back up in the Cloud: Keep backups of cloud workloads internal to the cloud
• Back up to the Cloud: Allow on premise workloads the ability to leverage AWS
• Disaster Recovery to the Cloud: Automate disaster recovery to the cloud on a scheduled basis
• Workload Portability: Rest assured that virtual servers can be moved from on-premise to the cloud and back, keep your data available wherever you need it
• Archiving to the Cloud: Moving legacy data to tier 2 storage in the cloud for long term archive
AWS and Commvault together combine to minimize networking, storage and infrastructure
costs, while providing the business a sound data protection and disaster recovery strategy.
Backup to AWS Approaches
Amazon S3
Amazon GlacierAWS
DirectConnect
InternetAmazon S3-IA
Applicationservers
Cloud Gateway
Local disk
MediaServer
Cloud Gateway
HTTPS/API
Applicationservers
Backup SW cloud connector
Local diskMedia
Server with cloud
connector
HTTPS/API
NetApp AltaVault Backup from On-premises to S3/Glacier
Common backup applications integrated with AltaVaultSolve backup & archive headaches with cloud-integrated storage
90% reduction in time, cost, and data volumes Shrink recovery times from days to minutes 85% of backup & software providers supported
Glacier
On Premises
AWS
Cloud-integrated storage appliance
NetApp AltaVault
FAS
E-SeriesNon-NetApp
Storage
NetApp SnapProtect Arcserve CommVault Simpana EMC NetWorker HP Data Protector IBM Tivoli
Storage Mgr
Symantec Backup Exec
Symantec NetBackup
Veeam Microsoft SQL
Server Oracle RMAN
S3
AltaVault also available on marketplace to protect cloud-native workloads
Seamlessly integrates into existing storage and backup software environment
Caches recent backups locally, vaults older copies to the cloud
Store data in the public or private cloud of choice
Deduplicates, compresses, and encrypts
© 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
unified file services that extend from endpoints, to remote
offices, to the cloud.
snapshots, file versioning and file sync runs across all access points via the cloud
data is secured and optimized at
the source
all stored in your AWS VPC, data is stored
on AWS S3-IA
Integrated with trusted enterprise security and
management tools
ROBO NAS Gateways
Endpoint Apps
Cloud Server Agents
Data ProtectionEngine
File SyncEngine
with centralized automation, management and multi-tenancy
Identitymanagement
datagovernance
cloudorchestration
S3 Infrequent Access
CTERA GlobalDeduplication
Ctera: Enterprise File Services Platform
Summary – When to Use each Service
IF YOU NEED: CONSIDER:An optimized or replacement Internet connection to:
connect directly into an AWS regional datacenter Direct Connect
migrate TB or PB of data to the cloud Snowball
Accelerate data transfer S3 Transfer Acceleration, CloudFront, AWS Partner
A friendly interface into S3 to:cache data locally in a hybrid model (for performance reasons) Storage Gateway, AWS Partner
redirect backups or archives with minimal disruption Storage Gateway, AWS Partner
aggregate data streams from multiple devices Kinesis Firehose
Thank you!