aws storage services for hybrid cloud - files.meetup.comfiles.meetup.com/19647895/aws storage...

31
© 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Alex Tomic, Solutions Architect March 23, 2016 AWS Storage Services for Hybrid Cloud

Upload: hadien

Post on 06-Feb-2018

221 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: AWS Storage Services for Hybrid Cloud - files.meetup.comfiles.meetup.com/19647895/AWS Storage Services for a Hybrid Cloud.… · AWS Storage Services for Hybrid Cloud . ... Hybrid

© 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Alex Tomic, Solutions Architect

March 23, 2016

AWS Storage Services

for Hybrid Cloud

Page 2: AWS Storage Services for Hybrid Cloud - files.meetup.comfiles.meetup.com/19647895/AWS Storage Services for a Hybrid Cloud.… · AWS Storage Services for Hybrid Cloud . ... Hybrid

Why AWS storage?

Page 3: AWS Storage Services for Hybrid Cloud - files.meetup.comfiles.meetup.com/19647895/AWS Storage Services for a Hybrid Cloud.… · AWS Storage Services for Hybrid Cloud . ... Hybrid

Introduction: Why choose AWS for storage

Compelling

Economics Easy to Use Reduce risk

Speed,

Agility, Scale

Pay as you go

No upfront investment

No commitment

No risky capacity

planning

No need to provision

for redundancy

or overhead

Self service

administration

SDKs for simple

integration

Durable and Secure

Avoid risks of physical

media handling

Reduce time to market

Focus on your

business, not your

infrastructure

Page 4: AWS Storage Services for Hybrid Cloud - files.meetup.comfiles.meetup.com/19647895/AWS Storage Services for a Hybrid Cloud.… · AWS Storage Services for Hybrid Cloud . ... Hybrid

AWS Global Infrastructure

12 Regions

33 Availability Zones

54 Edge locations

Control your geographic locality

for performance and compliance

Page 5: AWS Storage Services for Hybrid Cloud - files.meetup.comfiles.meetup.com/19647895/AWS Storage Services for a Hybrid Cloud.… · AWS Storage Services for Hybrid Cloud . ... Hybrid

Storage Choices in AWS

Amazon S3

Durable object storage

Amazon EBS

Block storage for use

with Amazon EC2

Amazon Glacier

Archival storage

Amazon EFS

File storage for use with

Amazon EC2

Amazon S3-IA

Infrequently accessed

data store

Block

Object

Page 6: AWS Storage Services for Hybrid Cloud - files.meetup.comfiles.meetup.com/19647895/AWS Storage Services for a Hybrid Cloud.… · AWS Storage Services for Hybrid Cloud . ... Hybrid

Amazon S3 and S3-Infrequent Access

Highly durable object storage for all types of data

Internet-scale storage

Grow without limits

Benefit from AWS’s

massive security

investments

Built-in redundancy

Designed for

99.999999999%

durability

Low price per GB

per month

No commitment

No up-front cost

Page 7: AWS Storage Services for Hybrid Cloud - files.meetup.comfiles.meetup.com/19647895/AWS Storage Services for a Hybrid Cloud.… · AWS Storage Services for Hybrid Cloud . ... Hybrid

Amazon Glacier

Archival storage for infrequently accessed data

Amazon Glacier

is optimized for

infrequent retrieval

Stop managing

physical media

Even lower cost than

Amazon S3;

Same high durability

Page 8: AWS Storage Services for Hybrid Cloud - files.meetup.comfiles.meetup.com/19647895/AWS Storage Services for a Hybrid Cloud.… · AWS Storage Services for Hybrid Cloud . ... Hybrid

Selecting the right Object Storage

S3

S3-IA

Glacier

L

i

f

e

c

y

c

l

e

Available S3: 99.99%

S3-IA: 99.9%

Performant Low Latency

High Throughput ≥ 30 Days ≥ 128K

≥ 90 Days

Durable 99.999999999%

Scalable Elastic capacity

No preset limits

> 0K $0.007/GB per month

$0.0125/GB per month

“Hot” Data Active and/or

Temporary Data

“Warm” Data Infrequently

Accessed Data

“Cold” Data Archive and

Compliance Data

≥ 0 Days > 0K $0.03/GB per month

3 – 5 Hrs

$0.01/GB retrieval

$0.01/GB retrieval < 5%

Page 9: AWS Storage Services for Hybrid Cloud - files.meetup.comfiles.meetup.com/19647895/AWS Storage Services for a Hybrid Cloud.… · AWS Storage Services for Hybrid Cloud . ... Hybrid

Why hybrid approach?

Page 10: AWS Storage Services for Hybrid Cloud - files.meetup.comfiles.meetup.com/19647895/AWS Storage Services for a Hybrid Cloud.… · AWS Storage Services for Hybrid Cloud . ... Hybrid

Why Hybrid IT

Leverage existing investments

Increase agility

Flatten learning curve

A

$

Page 11: AWS Storage Services for Hybrid Cloud - files.meetup.comfiles.meetup.com/19647895/AWS Storage Services for a Hybrid Cloud.… · AWS Storage Services for Hybrid Cloud . ... Hybrid

The Good News is that Cloud isn’t an ‘All or Nothing’

Choice

Corporate

Data Centers

On-Premises

Resources

Cloud

Resources Integration

Page 12: AWS Storage Services for Hybrid Cloud - files.meetup.comfiles.meetup.com/19647895/AWS Storage Services for a Hybrid Cloud.… · AWS Storage Services for Hybrid Cloud . ... Hybrid

Hybrid infrastructure services

AWS

Import/Export

Amazon Virtual

Private Cloud AWS Direct

Connect

Virtual Private

Network

Directory

Services

Identity & Access

Management CloudTrail Key Management

Service

Amazon

Route 53

VPC

peering

Page 13: AWS Storage Services for Hybrid Cloud - files.meetup.comfiles.meetup.com/19647895/AWS Storage Services for a Hybrid Cloud.… · AWS Storage Services for Hybrid Cloud . ... Hybrid

Hybrid architecture:

how do we connect to AWS?

Page 14: AWS Storage Services for Hybrid Cloud - files.meetup.comfiles.meetup.com/19647895/AWS Storage Services for a Hybrid Cloud.… · AWS Storage Services for Hybrid Cloud . ... Hybrid

VPC subnet

Availability Zone

Security group

VPC subnet

Availability Zone

Security group

Virtual Gateway

AWS Virtual Private Network (IPSec VPN)

o IPSec hardware VPN connection

Supported VPN appliances

o Encryption and Validation

o Private RFC 1918 Addressing

o Uses Border Gateway Protocol

(BGP) for routing and fail-over

o VPN Service provides managed

redundant end-points

Corporate data center

Users

Data center router

Servers

Internet

IPSec VPN

Page 15: AWS Storage Services for Hybrid Cloud - files.meetup.comfiles.meetup.com/19647895/AWS Storage Services for a Hybrid Cloud.… · AWS Storage Services for Hybrid Cloud . ... Hybrid

AWS Direct Connect

o Requires Layer 2 single mode fiber

1000BASE-LX or 10GBASE-LR

o Requires 802.1Q VLANs across

connection.

Tagging of IP traffic

o Routing uses BGP A/A or A/P

multipath.

o Each DX is mapped to a single

AWS Region

Corporate data center

Users

VPC subnet

Availability Zone

Security group

VPC subnet

Availability Zone

Security group

Data center router

Customer router

Servers

AWS Direct Connect location

AWS Direct Connect routers

Virtual Gateway

Page 16: AWS Storage Services for Hybrid Cloud - files.meetup.comfiles.meetup.com/19647895/AWS Storage Services for a Hybrid Cloud.… · AWS Storage Services for Hybrid Cloud . ... Hybrid

Hybrid architecture:

how to integrate authentication?

Page 17: AWS Storage Services for Hybrid Cloud - files.meetup.comfiles.meetup.com/19647895/AWS Storage Services for a Hybrid Cloud.… · AWS Storage Services for Hybrid Cloud . ... Hybrid

AWS Directory Service

Features

AWS Directory Service for Microsoft

Active Directory (Enterprise Edition)

Avoid complexity and cost of hosting

SAML-based federation infrastructure

Establish trust between on-premises AD

and AWS Directory Service

Requires IPSec VPN or Direct Connect connectivity

AWS Directory Service Connect

Corporate data center

Users

AD.Domain

Servers

Domain controller

VPC subnet

Availability Zone

Security group

Virtual Gateway

VPC subnet

Availability Zone

Security group

Page 18: AWS Storage Services for Hybrid Cloud - files.meetup.comfiles.meetup.com/19647895/AWS Storage Services for a Hybrid Cloud.… · AWS Storage Services for Hybrid Cloud . ... Hybrid

Active Directory DCs in your VPC

o Reduced back-reach Traffic

o Reduced Latency for Authentication

o Additional Resiliency

o Enablement of both:

Multi-Master Read/Write Domain Controllers

Read-only Domain Controllers (RODCs)

o Requires IPSec VPN or Direct Connect connectivity

Active Directory Replication

Corporate data center

Users

AD.Domain

Servers

Domain controller

Domain controller

VPC subnet

Availability Zone

Security group

Virtual Gateway

Domain controller

VPC subnet

Availability Zone

Security group Type Port Number

TCP 54, 88, 135, 137, 139, 389, 445, 464, 636, 3268, 3269, 5722, 49152-65535

UDP 53,67,123, 138, 389, 445, 464, 2535, 5355, 49152-65535

Replication

Page 19: AWS Storage Services for Hybrid Cloud - files.meetup.comfiles.meetup.com/19647895/AWS Storage Services for a Hybrid Cloud.… · AWS Storage Services for Hybrid Cloud . ... Hybrid

Enterprise Federation

Integrate identity management with AWS • Secure access to AWS resources using your IDM

• Provide SSO to AWS Management Console or API’s

• Build your own SSO federation using AWS STS service, or

• Federate with on-premise directories like Active Directory,

TFIM, OAM or another SAML 2.0 compliant IdP

Page 20: AWS Storage Services for Hybrid Cloud - files.meetup.comfiles.meetup.com/19647895/AWS Storage Services for a Hybrid Cloud.… · AWS Storage Services for Hybrid Cloud . ... Hybrid

Hybrid architecture: how to move data to AWS?

Page 21: AWS Storage Services for Hybrid Cloud - files.meetup.comfiles.meetup.com/19647895/AWS Storage Services for a Hybrid Cloud.… · AWS Storage Services for Hybrid Cloud . ... Hybrid

v

Import/Export Snowball

E-ink shipping label

Ruggedized

case

“8.5G Impact”

All data encrypted

end-to-end

Rain & dust

resistant

Tamper-resistant

case & electronics

50 TB

10Gb network

What about complex enterprise scenarios?

Page 22: AWS Storage Services for Hybrid Cloud - files.meetup.comfiles.meetup.com/19647895/AWS Storage Services for a Hybrid Cloud.… · AWS Storage Services for Hybrid Cloud . ... Hybrid

v

Amazon Storage Partner Ecosystem

Page 23: AWS Storage Services for Hybrid Cloud - files.meetup.comfiles.meetup.com/19647895/AWS Storage Services for a Hybrid Cloud.… · AWS Storage Services for Hybrid Cloud . ... Hybrid

v

Backup and archiving o Backup gateways integrated with

Amazon S3 o De-duplication

o Compression

o WAN Acceleration

o Leverage Amazon S3 archival to Amazon Glacier

Corporate data center

Amazon S3

Amazon S3-IA

Application server

Virtual server

File server

Database server

Backup system

Cloud Gateway

NetApp AltaVault cloud-integrated

storage appliance

AWS Marketplace Partners

Amazon Glacier

Page 24: AWS Storage Services for Hybrid Cloud - files.meetup.comfiles.meetup.com/19647895/AWS Storage Services for a Hybrid Cloud.… · AWS Storage Services for Hybrid Cloud . ... Hybrid

v

Storage Expansion

o Virtual volumes presented to local network iSCSI,

NFS and CIFS volumes

o Local disk cache to provide fast on-premise access

o Gateway side encryption for security

Corporate data center

Application server

Virtual server

File server

Storage appliance

Cloud Gateway

NetApp AltaVault cloud-

integrated storage appliance

AWS Marketplace Partners

Amazon S3

Amazon S3-IA

IAS

Amazon Glacier

Page 25: AWS Storage Services for Hybrid Cloud - files.meetup.comfiles.meetup.com/19647895/AWS Storage Services for a Hybrid Cloud.… · AWS Storage Services for Hybrid Cloud . ... Hybrid

v

Storage Replication

o Synchronize data with AWS EBS storage

o Perform byte-level replication by volume

o Virtual appliance in VPC for added networking

security and connectivity

Corporate data center

EC2 instance with EBS volumes

Application server

Virtual server

File server

Storage appliance

NetApp Cloud ONTAP

clustered storage operating system

AWS Marketplace Partners

EC2 Appl-iance

Page 26: AWS Storage Services for Hybrid Cloud - files.meetup.comfiles.meetup.com/19647895/AWS Storage Services for a Hybrid Cloud.… · AWS Storage Services for Hybrid Cloud . ... Hybrid

How do you use data in AWS?

Page 27: AWS Storage Services for Hybrid Cloud - files.meetup.comfiles.meetup.com/19647895/AWS Storage Services for a Hybrid Cloud.… · AWS Storage Services for Hybrid Cloud . ... Hybrid

ENTERPRISE APPS

DEVELOPMENT & OPERATIONS MOBILE SERVICES APP SERVICES ANALYTICS

Data Warehousing

Hadoop/Spark

Streaming Data Collection

Machine Learning

Elastic Search

Virtual Desktops

Sharing & Collaboration

Corporate Email

Backup

Queuing & Notifications

Workflow

Search

Email

Transcoding

One-click App Deployment

Identity

Sync

Single Integrated Console

Push Notifications

DevOps Resource Management

Application Lifecycle Management

Containers

Triggers

Resource Templates

TECHNICAL & BUSINESS SUPPORT

Account Management

Support

Professional Services

Training & Certification

Security & Pricing Reports

Partner Ecosystem

Solutions Architects

MARKETPLACE

Business Apps

Business Intelligence

Databases DevOps Tools

Networking Security Storage

Regions Availability Zones

Points of Presence

INFRASTRUCTURE

CORE SERVICES

Compute VMs, Auto-scaling, & Load Balancing

Storage Object, Blocks, Archival, Import/Export

Databases Relational, NoSQL, Caching, Migration

Networking VPC, DX, DNS

CDN

Access Control Identity Management

Key Management & Storage

Monitoring & Logs

Assessment and reporting

Resource & Usage Auditing

SECURITY & COMPLIANCE

Configuration Compliance

Web application firewall

HYBRID ARCHITECTURE

Data Backups

Integrated App Deployments

Direct Connect

Identity Federation

Integrated Resource Management

Integrated Networking

API Gateway

IoT

Rules Engine

Device Shadows

Device SDKs

Registry

Device Gateway

Streaming Data Analysis

Business Intelligence

Mobile Analytics

Page 28: AWS Storage Services for Hybrid Cloud - files.meetup.comfiles.meetup.com/19647895/AWS Storage Services for a Hybrid Cloud.… · AWS Storage Services for Hybrid Cloud . ... Hybrid

Case Study: Big Data Analytics

What Nasdaq needed • Replacement of on-premises legacy warehouse • Reduction of cost and increase in data capacity

Why they chose AWS (specifically Amazon Redshift) • Fulfillment of security and regulatory requirements • Cost efficiencies without sacrificing functionalities

Benefits realized • System that moves an average of 5.5 billion rows into

Amazon Redshift every day (with 14 billion on a peak day in Oct of 2014)

• Ability to increase accessibility of historic data to a growing number of internal groups

“The Nasdaq Group has been a user of Amazon Redshift since it was released and we are extremely happy with it…. Currently, our system is moving an average of 5.5 billion rows into Amazon Redshift every day.”

- Nate Simmons, Principal Architect

Page 29: AWS Storage Services for Hybrid Cloud - files.meetup.comfiles.meetup.com/19647895/AWS Storage Services for a Hybrid Cloud.… · AWS Storage Services for Hybrid Cloud . ... Hybrid

“As a new business within Broadridge, it’s important that we are able to reduce upfront costs while ensuring peak scalability. AWS’s ability to scale infinitely and provide a robust suite of services and capabilities meets our needs perfectly.”

- Robert Krugman, VP of Digital Strategy

What Broadridge needed • Technology platform to deliver investment industry content

to millions of consumers

Why they chose AWS • Ability to meet scale – documents from hundreds of

providers to millions of consumers • Fulfilment of security requirements to protect sensitive

information like monthly statements, trade confirmations, tax documents, and regulatory disclosures

Benefits realized • Inlet, the resulting platform, is delivering content through

Broadridge’s relationships with Financial Services companies

Case Study: Digital Innovation

Page 30: AWS Storage Services for Hybrid Cloud - files.meetup.comfiles.meetup.com/19647895/AWS Storage Services for a Hybrid Cloud.… · AWS Storage Services for Hybrid Cloud . ... Hybrid

“Using AWS helps us reduce a 10-day process to 10 minutes. That’s trans-formative: it broadens our ability to discover.”

- Peter Phillips, Managing Director

What Aon needed • Perform actuarial calculations with greater computing power • Information delivery within shorter time frames and less cost

Why they chose AWS • Ability to spin up large numbers of Graphical Processing Units

(or GPUs) quickly and inexpensively • Quick delivery of an entire environment and functionality

Benefits realized • By processing on AWS, recalculating policies takes minutes

rather than hours or days • Ability to deliver client solutions more quickly, with richer risk

assessments • Cost savings that are passed to the customer

Case Study: High Performance Computing (HPC)

Page 31: AWS Storage Services for Hybrid Cloud - files.meetup.comfiles.meetup.com/19647895/AWS Storage Services for a Hybrid Cloud.… · AWS Storage Services for Hybrid Cloud . ... Hybrid

“We are growing rapidly, and our capacity, availability, and resiliency requirements are constantly changing. We needed to be able to scale our infrastructure, but that was becoming difficult due to physical limitations and slow response times from our datacenters.”

- Andy Montgomery, Head of Division for IT Operations

and Solution Design

What St. James’s Place needed • Better infrastructure that could keep up with growth • Overcome siloes due to new and legacy systems Why they chose AWS • New functionalities including data warehousing and

electronic business processing system • Ability to move legacy systems and upgrade in the cloud

Benefits realized • Better management of growth, including ~50% increase in

client and associate traffic annually • Scalability to manage demand fluctuations • Faster recovery model

Case Study: Re-architecting for Growth