aws connectivity, vpc design and security pro tips

Agenda

Agenda •  AWS Connectivity

–  Direct Connect

What is AWS Direct Connect…

•  Dedicated, private pipes into AWS

•  Create private (VPC) or public interfaces to AWS

•  Cheaper data-out rates than Internet (data-in still free)

•  Consistent network performance compared to Internet

•  Multiple AWS accounts can share a connection

Why use AWS Direct Connect?

$0.000

$0.050

$0.100

$0.150

First 10TB Next 40TB

Next 100TB Next

350TB Direct Connect

Internet

Public Subnet

Availability Zone A

Private Subnet

Public Subnet

Availability Zone B

Private Subnet

Instance A 10.1.1.11 /24

Instance C 10.1.3.33 /24

Instance B 10.1.2.22 /24

Instance D 10.1.4.44 /24

VPC CIDR: 10.1.0.0 /16

Virtual Private

Gateway (VGW)

Internet Gateway

(IGW)

Only 1 IGW and 1 VGW per VPC

VPN connection Customer

data center Customer

data center

AWS Direct Connect

Route Table

Destination Target

10.1.0.0/16 local

Internal CIDR VGW

Direct Connect – Single Link, Single CGW

VPC 1 10.1.0.0/16 AWS Direct

Connect Customer Gateway

Colocation

DX Location

Customer Edge Router

172.16.0.0/16

Customers DC Customers

Subnet Service Provider

Network ` `

VGW AWS

Network DX POP Location

Cross Connect

Customer Gateway Router

Circuit to Customers

Network

Customers Network

Backbone

Circuit to Customers

Site

Customer Provider

Edge Router Customers

Local Network

Demarcation




Colocation

DX Location


172.16.0.0/16



Network ` `

From - To

CGW to VGW VGW to CGW

Route

172.16.0.0/16 10.1.0.0/16

Metric

- -

eBGP From - To

Customer to CGW CGW to Customer

Route

172.16.0.0/16 10.1.0.0/16

Routing – Probably eBGP

Layer 2 VLAN Connectivity

BGP is a requirement for Direct Connect: http://aws.amazon.com/directconnect/faqs/

VLAN Y

VLAN X

VIFs

virtual private cloud 1

virtual private cloud 2

virtual private cloud N

…

public endpoints

Region Direct Connect Location

private VIF 1 public virtual interface (VIF)

private VIF 2 VLAN Z

VLAN N

AWS DX Router

Customer Router

Each interface can be associated with a different

AWS Account. (Hosted Virtual Interfaces)

Public Virtual Interfaces (VIFs)

Private Virtual Interfaces (VIFs)


–  VPN –  Design Patterns

Customer Network

Only 1 IGW and 1 VGW per VPC

Customer Network


–  VPN –  Design Patterns




Colocation

DX Location


172.16.0.0/16



Network ` `

With IPSEC Failover




Colocation

DX Location


172.16.0.0/16



Network ` `

IPSEC over The Internet

From - To


Route

172.16.0.0/16 10.1.0.0/16

Metric

- -

eBGP

With IPSEC Failover

Direct Connect – Dual Links, Single CGW



Colocation

DX Location


172.16.0.0/16



Network ` `


VPC 1 10.1.0.0/16

AWS Direct Connect

Customer Gateway

Colocation

DX Location


172.16.0.0/16



Network ` `


VPC 1 10.1.0.0/16

AWS Direct Connect

Customer Gateway

Colocation

DX Location


172.16.0.0/16



Network ` `

From - To


Route

172.16.0.0/16 10.1.0.0/16

Metric LP 150

eBGP

From - To


Route

172.16.0.0/17 10.1.0.0/16

Metric LP 90

eBGP

-  You can split your route advertisements to the VGW -  Instead of using AS Path Prepend

CGW to VGW 172.16.128.0/17

Direct Connect – Dual Links, Dual CGW

VPC 1 10.1.0.0/16

AWS Direct Connect

Customer Gateway

Colocation

DX Location


172.16.0.0/16



Network ` `


VPC 1 10.1.0.0/16

AWS Direct Connect

Customer Gateway

Colocation

DX Location


172.16.0.0/16


Subnet

`

Service Provider Network

`


VPC 1 10.1.0.0/16

AWS Direct Connect

Customer Gateway

Colocation

DX Location


172.16.0.0/16


Subnet

` `

`



VPC 1 10.1.0.0/16

AWS Direct Connect

Customer Gateway

Colocation

DX Location


172.16.0.0/16


Subnet

`

`


`


VPC 1 10.1.0.0/16

AWS Direct Connect

Customer Gateway

Colocation

DX Location


172.16.0.0/16


Subnet

`

`


`

BGP AS - Y

BGP AS - X iBGP between Routers iBGP between Routers


VPC 1 10.1.0.0/16

AWS Direct Connect

Customer Gateway

Colocation

DX Location


172.16.0.0/16


Subnet

`

`


`

- So far so good? - What’s wrong with this topology?

- SPoF!


VPC 1 10.1.0.0/16

AWS Direct Connect

Customer Gateway

Colocation

DX Location


172.16.0.0/16

Customers DC Customers Subnet

`

`


`

Direct Connect – Dual Locations, Dual Links

VPC 1 10.1.0.0/16

AWS Direct Connect

Customer Gateway

Colocation

DX Location - 1


172.16.0.0/16


Subnet

`

`


`

AWS Direct Connect

Customer Gateway

Colocation

`

DX Location - 2

VPC 1 10.1.0.0/16

AWS Direct Connect

Customer Gateway

Colocation

DX Location - 1


172.16.0.0/16


Subnet

`

`

`

AWS Direct Connect

Customer Gateway

Colocation

`

DX Location - 2

DirectConnect – Dual Locations, Dual Links, Dual Routers


Multi Account DX

VPC 1 10.1.0.0/16

Customer Gateway

VLAN320 Private VI

AWS Direct Connect

Ethernet Trunk

SVI/Sub 320

IP 169.x.x.2 IP 169.x.x.1

BGP AS 65xxx BGP AS 17493

VLAN 320

Colocation

Multi-Account Direct Connect

Customer Gateway

VLAN320

Ethernet Trunk

SVI/Sub 320

IP 169.x.x.2

BGP AS 65xxx

VPC 1 10.1.0.0/16

Private VI

IP 169.x.x.1

BGP AS 17493

VLAN 320

Colocation

VPC 2 10.2.0.0/16 IP 169.y.y.1

BGP AS 17493

VLAN 330

VLAN330

SVI/Sub 330

IP 169.y.y.2

BGP AS 65xxx

Private VI

AWS Direct Connect

AWS Account 1

Multi-Account Direct Connect

Customer Gateway

VLAN320

Ethernet Trunk

SVI/Sub 320

IP 169.x.x.2

BGP AS 65xxx

VPC 1 10.1.0.0/16

Private VI

IP 169.x.x.1

BGP AS 17493

VLAN 320

Colocation

VPC 2 10.2.0.0/16 IP 169.y.y.1

BGP AS 17493

VLAN 330

VLAN330

SVI/Sub 330

IP 169.y.y.2

BGP AS 65xxx

Private VI

AWS Direct Connect

AWS Account 1

AWS Account 2

How to Delegate VI to Another Account. Step 1.

Delegate Virtual Interface to Another Account.

Step 2.

Agenda –  VPN –  Design Patterns

•  VPC Design –  Concepts

Avai

labi

lity

Zone

A

Avai

labi

lity

Zone

B

VPC A - 10.0.0.0/16

Avai

labi

lity

Zone

A

Avai

labi

lity

Zone

B

Choose your VPC address range •  Your own private, isolated

section of the AWS cloud •  Every VPC has a private IP

address space •  That maximum CIDR block

you can allocate is /16 •  For example 10.0.0.0/16 –

this allows 256*256 = 65,536 IP addresses

Select IP addressing strategy •  You can’t change the VPC

address space once it’s created

•  Think about overlaps with other VPCs or existing corporate networks

•  Don’t waste address space, but don’t’ constrain your growth either

VPC A - 10.0.0.0/16

Avai

labi

lity

Zone

A

VPC A - 10.0.0.0/16

Avai

labi

lity

Zone

A

10.0.1.0/24

10.0.2.0/24

10.0.3.0/24 10.0.5.0/24 10.0.4.0/24

VPC A - 10.0.0.0/16

Avai

labi

lity

Zone

A

10.0.1.0/24

10.0.2.0/24

10.0.3.0/24

EC2

10.0.5.0/24 10.0.4.0/24

EC2 App

Log

EC2 Web

Bastion

VPC A - 10.0.0.0/16

Avai

labi

lity

Zone

A

10.0.1.0/24

10.0.2.0/24

10.0.3.0/24

EC2

10.0.5.0/24 10.0.4.0/24

EC2 App

“Web servers can connect to app servers on port 8080”

Log

EC2 Web

Bastion

VPC A - 10.0.0.0/16

Avai

labi

lity

Zone

A

10.0.1.0/24

10.0.2.0/24

10.0.3.0/24

EC2

10.0.5.0/24 10.0.4.0/24

EC2 App


“Allow outbound

connections to the log server”

Log

EC2 Web

Bastion

VPC A - 10.0.0.0/16

Avai

labi

lity

Zone

A

10.0.1.0/24

10.0.2.0/24

10.0.3.0/24

EC2

10.0.5.0/24

Bastion

10.0.4.0/24

EC2 App


“Allow outbound

connections to the log server”

“Allow SSH and ICMP from

instances in the Bastion security

group”

Log

EC2 Web

VPC A - 10.0.0.0/16

Avai

labi

lity

Zone

A

10.0.1.0/24

10.0.2.0/24

10.0.3.0/24

EC2

10.0.5.0/24 10.0.4.0/24

EC2 App

Log

EC2 Web

Security groups •  Operate at the instance level •  Supports ALLOW rules only •  Are stateful •  Max 50 rules per security group •  Max 5 groups per instance

Bastion

VPC A - 10.0.0.0/16

Avai

labi

lity

Zone

A

10.0.1.0/24

10.0.2.0/24

10.0.3.0/24

EC2

Router 10.0.5.0/24 10.0.4.0/24

EC2 App

Log

EC2 Web

Bastion

VPC A - 10.0.0.0/16

Avai

labi

lity

Zone

A

10.0.1.0/24

10.0.2.0/24

10.0.3.0/24

EC2

Router 10.0.5.0/24 10.0.4.0/24

EC2 App

Log

EC2 Web

“Deny all traffic between the web server subnet and the database server subnet”

Bastion

VPC A - 10.0.0.0/16

Avai

labi

lity

Zone

A

10.0.1.0/24

10.0.2.0/24

10.0.3.0/24

EC2

Router 10.0.5.0/24 10.0.4.0/24

EC2 App

Log

EC2 Web

Bastion

NACLs are optional •  Applied at subnet level •  Stateless and permit all by default •  ALLOW and DENY •  Applies to all instances in the subnet •  Use as guard rails (port 21, 135,…)

VPC A - 10.0.0.0/16

Avai

labi

lity

Zone

A

10.0.1.0/24

10.0.2.0/24

10.0.3.0/24

EC2

Router 10.0.5.0/24 10.0.4.0/24

EC2 App

Log

EC2 Web EC

2 Web

Elastic Load Balancer

Bastion

VPC A - 10.0.0.0/16

Avai

labi

lity

Zone

A

10.0.1.0/24

10.0.2.0/24

EC2

10.0.3.0/24

EC2

Router 10.0.5.0/24 10.0.4.0/24

EC2 App

Log

EC2 Web EC

2 Web EC

2 EC2 Web

Elastic load balancers •  Instances can automatically be

added and removed from the balancing pool using rules

•  You can add instances into security groups at launch time

Elastic Load Balancer

Auto scalin

g

Bastion

VPC A - 10.0.0.0/16

Avai

labi

lity

Zone

A

EC2

EC2

EC2 App

EC2 Web EC

2 Web EC

2 EC2 Web

Internet Gateway

VPC Router

VPC A - 10.0.0.0/16

Avai

labi

lity

Zone

A

EC2

EC2

EC2 App

EC2 Web EC

2 Web EC

2 EC2 Web

Internet Gateway

VPC Router

Internet routing •  Add route tables to subnets to

control Internet traffic flows – these become Public subnets

•  Internet Gateway routing allows you to allocate a static Elastic IP address or use AWS-managed public IP addresses to your instance

VPC A - 10.0.0.0/16

Avai

labi

lity

Zone

A

EC2

EC2

EC2 App

EC2 Web EC

2 Web EC

2 EC2 Web

Internet Gateway

VPC Router

Internet routing •  Use a NAT instance to

provide Internet connectivity for private subnets - required to access AWS update repositories

•  This will also allow back-end servers to route to AWS APIs – for example storing logs on S3, or using Dynamo, SQS, SNS and SWS

NAT

VPC A - 10.0.0.0/16

Avai

labi

lity

Zone

A

10.0.1.0/24

10.0.2.0/24

EC2

10.0.3.0/24

EC2

NAT

10.0.4.0/24

EC2 App

EC2 Web EC

2 Web EC

2 EC2 Web VPC

Router

Amazon S3

DynamoDB

Amazon SNS

Amazon SQS

Internet Gateway

NAT


•  VPC Design –  Concepts –  Design Patterns

To NACL or not to NACL? Pros

Another layer of defense Can speed up deals

•  Fits legacy IT models •  Network/FW Engineer’s friend

Can help with networking compliance •  Separate groups for SGs/NALCs

Explicit deny rules Apply to an entire subnet

Cons Adds complexity Can slow down adoption

•  Fits legacy IT processes •  DevOps Enemy

Potentially not necessary for compliance

•  Third-party proactive controls •  SG audits (programmable infra)

Stateless FW rules Apply only to subnets/CIDR addresses

NACL Best Practices

Routing Instances Love Them

NAT instances VPN tunnels (between VPCs) Data loss prevention

Intrusion detection

Hate Them Single point of failure Extra costs (EC2, third-party licenses) More for customer to manage

Potential network bottleneck

Routing Instance Best Practices

AWS region

Public-facing web app

Internal company app

What’s next?

VPN connection

Customer data center

Multiple VPCs

Multiple VPCs use case

Multiple VPCs tips and tricks


Internal company app #2

HA pair VPN endpoints Customer data center




Internal company Dev

Internal company QA

AWS region

Backup AD, DNS Monitoring Logging

Multiple VPCs over IPSEC VPN

About IPSEC and multiple VPCs



HA pair VPN endpoints Customer data center





Internal company QA

AWS region

Backup AD, DNS Monitoring Logging

Multiple VPCs over AWS Direct Connect

Direct Connect Facility

Customer Data Center

Physical Connection

Logical Connections

VLANs

Logical Connections

VLANs

About AWS Direct Connect and multiple VPCs

•  Security groups and NACLs still apply

AWS region



HA pair VPN endpoints

company data center




Services VPC


Internal company QA

AD, DNS

Monitoring Logging

•  Security groups still bound to single VPC

Multiple VPCs over VPC Peering

VPC peering use cases

10.1.0.0/16

10.0.0.0/16

•  VPCs within same region Peer

Request

Peer Accept

•  Same or different accounts

•  IP space cannot overlap

•  Only 1 between any 2 VPCs

VPC peering configuration

10.1.0.0/16

10.0.0.0/16 10.0.0.0/16

✔

Overlapping IP is not a dead end

10.0.0.0/16 10.0.0.0/16

PCX-1 PCX-2

Subnet 1

10.1.1.0/24 Subnet 2

10.1.2.0/24

10.1.0.0/16 Route Table Subnet 1

Destination Target

10.1.0.0/16 local

10.0.0.0/16 PCX-1

Route Table Subnet 2

Destination Target

10.1.0.0/16 local

10.0.0.0/16 PCX-2

A

B C

10.1.0.0/16

10.4.0.0/16 10.0.0.0/16

10.3.0.0/16

172.16.0.0/16 192.168.0.0/16

10.2.0.0/16

172.17.0.0/16

C A

10.1.0.0/16

10.4.0.0/16 10.0.0.0/16

10.3.0.0/16

172.16.0.0/16 192.168.0.0/16

10.2.0.0/16

172.17.0.0/16

company data center

10.10.0.0/16

10.4.0.0/16 10.0.0.0/16

172.16.0.0/16 192.168.0.0/16 172.17.0.0/16

10.1.0.0/16 10.2.0.0/16 10.3.0.0/16

Peer review

•  Shared infrastructure services moved to VPC

•  1 to 1 peering = app isolation

•  Security groups and NACLs still apply

AWS region



HA pair VPN endpoints

company data center




Services VPC


Internal company QA

AD, DNS

Monitoring Logging •  Security groups still bound to

single VPC

Multiple accounts

About VPC peering and multiple VPCs

Model 1: “Lollipop”

Model 2: “Shared Services Model”

Model 3: “HIPS Model”

Scenario #4 – “Threat Layer Model”

Model 5: “NIDS Model”

Model 6: “Hybrid Model”

S3 VPC endpoint


•  VPC Design –  Concepts –  Design Patterns

•  Security Pro Tips

IAM Best Practices

AWS Trusted Advisor

CloudTrail – Log & monitor these!

•  API actions with potential impact –  Internet Gateway –  Routes and Route Tables –  Network ACLs –  EC2 instances (run/create/launch/terminate) –  Security Groups –  CloudTrail (stop/delete/update) –  Put[Group/Role/User]Policy –  ModifyAccount –  ModifyBilling, ModifyPaymentMethods –  "Type":"Root" –  Create[User/Role/Group] –  CreateAccessKey

Continuous Change Recording Changing Resources

AWS Config History

Stream

Snapshot (ex. 2014-11-05) AWS Config

ENCRYPT YOUR SENSITIVE DATA

segregate duties

With AWS IAM you get to control who can do what in your AWS environment and from where Fine-‐grained control of your AWS cloud with two-‐factor authen;ca;on Integrated with your exis;ng corporate directory using SAML 2.0 and single sign-‐on

AWS account owner

Network management

Security management

Server management

Storage management

DDoS Protection

Inbound HTTP

CloudFront

Amazon S3

WAF Dynamic

App

App

App Peering

DDoS

users

Instance patching and upgrades

VPC Flow Logs

Discussion…