arquitetura hibrida - integrando seu data center com a nuvem da aws

Michel Pereira

Hybrid Architecture: Integrating your Data Center with AWS

Enterprise Solutions Architect

Vários Tutoriais , treinamentos e mentoria em

português

Inscreva-se agora !!

http://awshub.com.br

http://awshub.com.br/



Datacenters limitations

How much time to deliver a project?

Capacity

Datacenter

Agility

Datacenter

Requirement Server Load Balancer Firewall Storage

Approvals Hardware or VM aquisition Capacity allocation Ticket queuing Provisioning Configuration

Request

Availability!!!

Capacity is a premium resource...

...and take time to have!

Cost

Datacenter

Project Two Potential impact: HIGH Potential cost: HIGH

Project One Potential impact: LOW Potential cost: HIGH

Project Three Potential impact: LOW Potential cost: LOW

DENIED

DENIED

APPROVED

Cost of infrastructure can inhibit innovation

And now comes an era of an Elastic Data Center

Flexible, on-demand facilities

Cost

AWS

Project Two Potential impact: HIGH Potential cost: HIGH

Project One Potential impact: LOW Potential cost: HIGH

Project Three Potential impact: LOW Potential cost: LOW

APPROVED

APPROVED

APPROVED COMPLETED

COMPLETED

COMPLETED

And now comes an era of an Elastic Data Center

Flexible, on-demand facilities

That you can integrate with your on-premises

No pain, only gain

Regions

Where?

US-WEST (Oregon) EU-WEST (Ireland)

ASIA PAC (Tokyo)

ASIA PAC

(Singapore)

US-WEST (N. California)

SOUTH AMERICA (Sao Paulo)

US-EAST (Virginia)

AWS GovCloud (US)

ASIA PAC (Sydney)

Regions

US-WEST (Oregon)) EU-WEST (Ireland)

ASIA PAC (Tokyo)

ASIA PAC

(Singapore)

US-WEST (N. California)

SOUTH AMERICA (Sao Paulo)

US-EAST (Virginia)

AWS GovCloud (US)

ASIA PAC (Sydney)

Availability Zones

Amazon Virtual Private Cloud (Amazon VPC) enables you to

launch Amazon Web Services (AWS) resources into a virtual

network that you’ve defined.

1 What is

Amazon

VPC?

Bridge your VPC and your

onsite IT infrastructure with an

encrypted VPN connection

Store data in S3 and set

permissions to allow access

only from within your VPC

Assign multiple IP address and

attach multiple ENIs and EIPs

to EC2 instances

Control Inbound and outbound

access to and from individual

subnets

Specify your own private IP

address range from any

ranges you choose

Divide your private IP address

range into one or more public

or private subnets.

What is VPC?

Bridge your VPC and your

onsite IT infrastructure with an

encrypted VPN connection

Store data in S3 and set

permissions to allow access

only from within your VPC

Assign multiple IP address and

attach multiple ENIs and EIPs

to EC2 instances

Control inbound and outbound

access to and from individual

subnets

Specify your own private IP

address range from any

ranges you choose

Divide your private IP address

range into one or more public

or private subnets.

Only cost is

optional hardware

VPN

What is VPC?

Overview of VPC

Public Subnet

10.0.0.0/24

Internet Customer Network 172.16.0.0/8

Web Server

10.0.0.5

198.51.100.1 (EIP)

Web Server

10.0.0.6

198.51.100.2 (EIP)

Web Server

10.0.0.7

198.51.100.3 (EIP)

NAT

10.0.0.8

198.51.100.4 (EIP)

Private Subnet

10.0.1.0/24

DB Server

10.0.1.5

DB Server

10.0.1.6

DB Server

10.0.1.7

Amazon EC2

API endpoint

Amazon S3

API endpoint

R VPC

10.0.0.0/16

Custom Route Table

Destination Target

10.0.0.0/16 local

0.0.0.0/0 Internet Gateway

Main Route Table

Destination Target

10.0.0.0/16 local

172.16.0.0/8 Virtual Private Gateway

0.0.0.0/0 NAT Instance

Internet Gateway

VPN Gateway

With Amazon VPC, you can define a virtual network topology that

closely resembles a traditional network that you might operate in

your own data center.

2 Subnets,

gateways,

and routes

• Select a region for your own network in the cloud.

Availability Zone Availability Zone Availability Zone Availability Zone

Insert network here.


• Create your own subnet (s).


VPC Subnet 1 VPC Subnet 2 VPC Subnet n


• Create your own subnet (s).

• Configure custom routing rule(s).


VPC Subnet 1 VPC Subnet n

• Availability zone boundary

• Routing table boundary

• Network access control boundary


VPC Subnet 1

VPC Subnet 4

VPC Subnet 3

VPC Subnet 2

• Routing traffic

• Virtual Private Gateway

• Internet Gateway

Availability Zone

VPC Subnet 1

VPC Subnet 2

Corporate Data center

• Virtual Private Gateway (VGW)

• IPSEC VPN

Availability Zone

VPC Subnet 1

VPC Subnet 2

VPN Gateway Route Table Route Table Corporate Data center


• IPSEC VPN

• AWS Direct Connection

Availability Zone

VPC Subnet 1

VPC Subnet 2

AWS Direct

Connect

VPN Gateway Route Table

Route Table Corporate Data center

• Internet Gateway (IGW)

• IPSEC VPN

Availability Zone

VPC Subnet 1

VPC Subnet 2

VPN Gateway Route Table Route Table

Route Table Internet Gateway



• Internet Gateway (IGW)

Availability Zone

VPC Subnet 1

VPC Subnet 2

VPN Gateway Route Table Route Table

Route Table Internet Gateway


Good, got my VPC. And?

Welcome to the cloud, I’ll be your guide

Amazon Elastic Compute Cloud (EC2)

• Resizable compute capacity

• Complete control of your computing resources

• Reduces the time required to obtain and boot new server instances to minutes

• Scale capacity as your computing requirements change

• Pay only for capacity that you actually use

EC2 Instances Family Use

Standard Have memory-to-CPU ratios suitable for most general-purpose applications.

Micro Provide a small amount of consistent CPU resources and enable you to burst CPU capacity when additional cycles are available. They're well-suited for lower throughput applications and websites that consume significant compute cycles periodically.

High Storage Provide very high storage density and high sequential read and write performance per instance. They are well-suited for data warehousing, Hadoop/MapReduce, and parallel file systems.

High Memory Have proportionally more memory resources. They're well suited for high-throughput applications, such as database and memory caching applications.

High Memory Cluster Have large amounts of memory coupled with high CPU and network performance. These instances are well suited for in-memory analytics, graph analysis, and scientific computing applications.

High I/O Provide tens of thousands of low-latency, random I/O operations per second (IOPS) to an application. They're well-suited for NoSQL databases, clustered databases, and OLTP (online transaction processing) systems.

High CPU Have proportionally more CPU resources than memory (RAM). They're well-suited for compute-intensive applications.

Cluster Compute Have a very large amount of CPU coupled with increased networking performance. They're well-suited for High Performance Compute (HPC) applications and other demanding network-bound applications.

Cluster GPU Provide general-purpose graphics processing units (GPUs), with proportionally high CPU and increased network performance for applications that benefit from highly parallelized processing. They're well-suited for HPC applications as well as rendering and media processing applications.

Standard

Name Memory Compute Units

Virtual Cores Instance Store Volumes

Architecture I/O Performance

Available for Spot Instance

API Name

M1 Extra Large 15 GiB 8 4 (with 2 ECUs

each)

1680 GB (4 x

420 GiB)

64-bit High Yes m1.xlarge

M1 Large 7.5 GiB 4 2 (with 2 ECUs

each)

840 GiB (2 x

420 GiB)

64-bit Moderate Yes m1.large

M1 Medium 3.75 GiB 2 1 400 GiB (1 x

400 GiB)

32-bit and 64-bit Moderate Yes m1.medium

M1 Small 1.7 GiB 1 1 150 GiB (1 x

150 GiB)

32-bit and 64-bit Moderate Yes m1.small

M3 Double Extra

Large

30 GiB 26 8 (with 3.25 ECUs

each)

EBS storage only 64-bit High Yes m3.2xlarge

M3 Extra Large 15 GiB 13 4 (with 3.25 ECUs

each)

EBS storage only 64-bit Moderate Yes m3.xlarge

Micro and High CPU


Virtual Cores Instance Store

Volumes


Available for Spot

Instance

API Name

Micro 615 MiB Up to 2 (for short

periodic bursts)

1 None (use Amazon EBS

volumes for storage)

32-bit and 64-bit

Low Yes t1.micro





API Name

High-CPU Extra Large

7 GiB 20 8 (with 2.5 ECUs each)

1680 GiB (4 x 420 GiB)

64-bit High Yes c1.xlarge

High-CPU Medium

1.7 GiB 5 2 (with 2.5 ECUs each)

340 GiB (1 x 340 GiB)

32-bit and 64-bit

Moderate Yes c1.medium

High Storage and High I/O





API Name

High Storage Eight Extra Large

117 GiB 35 16 (8 cores + 8 hyperthreads)

48 TiB (24 x 2 TiB hard disk drives)

64-bit Very high (10 Gbps Ethernet)

No hs1.8xlarge



Volumes


Available for Spot

Instance

API Name

High I/O Quadruple Extra Large***


2 TiB (2 x 1 TiB SSD)

64-bit Very high (10 Gbps Ethernet)

No hi1.4xlarge

High Memory





API Name

High-Memory Double Extra Large


840 GiB (1 x 840 GiB)

64-bit High Yes m2.2xlarge

High-Memory

Extra Large

17.1 GiB 6.5 2 (with 3.25 ECUs each)

410 GiB (1 x 410 GiB)

64-bit Moderate Yes m2.xlarge

High-Memory Quadruple Extra Large


1680 GiB (2 x 840 GiB)

64-bit High Yes m2.4xlarge

Cluster

Name Memory Compute

Units


Volumes

Architecture I/O Performance Available for

Spot Instance

API Name

Cluster

Compute Eight Extra Large

60.5 GiB 88 16 (2 x Intel Xeon E5-

2670, eight-core with hyperthread)

3360 GiB (4 x

840 GiB)

64-bit Very high (10

Gbps Ethernet)

Yes cc2.8xlarge

Cluster

Compute Quadruple

Extra Large

22.5 GiB 33.5 8 (2 x Intel Xeon

X5570, quad-core with hyperthread)

1690 GiB (2 x

840 GiB)


Gbps Ethernet)

Yes cc1.4xlarge

High-Memory Cluster

High-Memory

Cluster Eight Extra Large

244 GiB 88 16 (2 x Intel Xeon E5-

2670, eight-core)

240 GiB (2 x

120 GiB SSD)


Gbps Ethernet)

Yes cr1.8xlarge

Cluster GPU

Cluster GPU

Quadruple Extra Large**

22.5 GiB (see

note after this table)

33.5 8 (2 x Intel Xeon

X5570, quad-core with hyper thread), plus 2

NVIDIA Tesla M2050

GPUs

1680 GiB (2 x

840 GiB)


Gbps Ethernet)

Yes cg1.4xlarge

1:Many Relationship Between AMIs and Instances

Amazon Machine Image

(AMI)

Instance

Server/Instance States

Traditional Hardware-Based Server States Amazon EC2 AMI

Amazon EC2 Instance States

Physical Interface

Firewall

Hypervisor

Smal

l

Larg

e

…

Smal

l

A Physical Host Has Multiple VMs

Virtual Machines

Physical Host

Auto Scaling

• Scale your Amazon EC2 capacity automatically

• Well suited for applications that experience variability in usage

• Available at no additional charge

Auto Scaling

• Automatically Scale Server Farms – Scale up and down

– (Re)Balance Across AZs

– Add/Remove from ELB if applicable

• Set a Thermostat – Don’t manage the furnace burners

Types of scaling

• Manual – Send an API call or use CLI to launch/terminate instances

– Only need to specify capacity change (+/-)

• By Schedule – Scale up/down based on date and time

• By Policy – Scale in response to changing conditions, based on user configured real-

time monitoring and alerts

• Automatic Rebalance – Instances are automatically launched/terminated to ensure the application is

balanced across multiple AZs

Auto Scaling

myASG

Launch Configuration

ami-0535d66c

sa-east-1a sa-east-1b

myELB

myLC

Auto Scaling


ami-0535d66c


myELB

Trigger: CPULoad Measure (M) : Average CPUUtilization Scale-out by 1 if M > 80% for 5 minutes Scale-in by 1 if M < 40% for 20 minutes

myASG

myLC

Auto Scaling


ami-0535d66c


myELB

Trigger: CPULoad Measure (M) : Average CPUUtilization Scale-out by 1 if M > 80% for 5 minutes Scale-in by 1 if M < 40% for 20 minutes

myASG

myLC

as-create-auto-scaling-group myASG –launch-configuration myLC –availability-zones sa-east-1a, sa-east-1b

–min-size 1 –max-size 10 –desired-capacity 3 –load-balancers myELB

In this case the auto scaling group knows about the ELB myELB because it was identified when the auto scaling group was created so the new instance is added to the ELB myELB pool

Anti-Pattern: Vertical

Vertical scaling (more CPU, memory, etc.) will eventually run out of room.

Auto Scaling Patterns

Pattern: Horizontal

Add and remove instances as needed

Auto Scaling Patterns

• O Magazine Luiza é uma das maiores redes varejistas com foco em bens duráveis e grande presença nas classes populares do Brasil. Conta com uma base de 30 milhões de clientes cadastrados, sendo 30% deles ativos.

• Há 2 anos criamos o luizalabs - braço de Pesquisa e Desenvolvimento do Magazine Luiza. Com o objetivo de criar novos produtos e servir de base de conhecimento para novas tecnologias.

“Plataforma mobile que suporta o dobro do volume de acessos,

gera quase 50% mais conversões - com apenas 15% do valor

gasto anteriormente”

“Escolhemos a AWS por 4 motivos:

Performance, Flexibilidade, Segurança

e Preço.” - André Fatala

O Desafio

• Substituir a primeira versão do site mobile do Magazine Luiza lançado em Dezembro de 2011 e hospedado em uma parceira da produtora.

• Resolvemos desenvolver (in house) a segunda versão do site mobile, objetivando uma aplicação mais clean, facilidade de navegação e, principalmente, uma navegação mais rápida devido aos problemas de conexão que enfrentamos com operadoras no Brasil.

• Escalabilidade - já que temos um canal que cresceu em média 98% comparado ao mesmo período do ano passado.

Sobre o Papel da AWS e Benefícios

alcançados

• Reduzimos a despesa com hospedagem em 85%

• Carregamento de páginas 3x mais rápida que originou uma taxa de rejeição 53% menor e um aumento na taxa de conversão de 36%

• A infraestrutura da AWS com seus recursos programáveis, permitiu que a nova plataforma fosse desenvolvida e colocada em produção em apenas 1 mês

Helping Enterprises with Hybrid IT Architectures

Most enterprises will run a hybrid IT architecture

Some workloads will run on-premises

Some workloads will run in the cloud

Goal is to make management & integration easy

Workloads can be migrated back and forth

Our “Hybrid” Focus: Tools to Support Hybrid IT

Architectures

Private Connections

Workload Migrations

Access Control Integration

Work with Existing

Management Tools

On-Premises Apps

Your Data Centers

Cloud Apps

Active Directory

VMware Images

Network Configuration

Your Data

Your On-Premises Apps Your Data Centers

Users & Access Rules

VM Import/Export

Your Private Network

Our Storage

Your Cloud Apps

Direct Connect VPC

IAM

Storage Gateway

Our “Hybrid” Focus: Tools to Support Hybrid IT Architectures

Cloud benefits

Run cloud, run

Amazon RDS

RDS is a fully managed relational database service that is

simple to deploy, easy to scale, reliable and cost-effective

Ease of Deployment and Patching

Push Button Scalability

Choice of DB Engines and Application Compatibility

Automated Backups and Disaster Recovery

User Snapshots and Cloning

Monitoring and Automatic Host Replacement

Petabyte-Scale Data Warehousing Feature Details

Optimized for Data Warehousing

Redshift uses a variety of innovations to obtain very high query performance on datasets ranging in size from hundreds of gigabytes to a petabyte or more.

Scalable Easily scale the number of nodes in your data warehouse up or down as your performance or capacity needs change

Fault tolerant Data replicated across Availability Zones.

Monitoring Integrated to CloudWatch.

Secure Encrypt data in transit and at rest. Can also be run in VPC to isolate your data warehouse cluster.

S3 intergration Loads data in parallel to each node from S3.

Elastic MapReduce Integrates with ERM via Data Pipeline.

On -premises

On -premises

AWS Cloud

AWS Cloud

DISASTER RECOVERYFOR LOCAL APPLICATIONS

Amazon

EC2

Amazon

VPC

Amazon

S3

AWS Storage Ga teway

AWS

Refere

nce

Archite

cture

s

Amazon

EBS

AmazonS3

AmazonVPC

AWS

Storage

Ga teway

VPC

Gateway

VPC

Gateway

Corpora te

User

Database

Server

(Reco

very)Data

base

Server

(Reco

very)

Appl icatio

n

Server

(Reco

very)Appl ic

ation

Server

(Reco

very)

4

3

Internet

Ga teway

Snapshots

AMIs

Files

Storage Volumes

SecureConnection

SecureConnection

Amazon

EC2

Amazon

EC2

2

1

5

Oracle S

ecure

Backups

Secure

Connection

Secure

Connection

Applicatio

n

Server

(Product ion)Applic

ation

Server

(Product ion)

Database

Server

(Product i

on)Database

Server

(Product i

on)

Corpora te

Data Center

Data Resto

ration

Amazon

EBS

Amazon

EBS

5

6

Michel Pereira

Hybrid Architecture: Integrating your Data Center with AWS

[email protected]

arquitetura hibrida - integrando seu data center com a nuvem da aws

Real Estate

vpc subnet n

overview of vpc public

eip private subnet

high potential cost

low potential cost

potential impact

private subnets

cost aws project