NexusBazaarvoice Cloud Infrastructure

Victor TracCloud Architect, Bazaarvoice

welcome to cloud computing!!1

BazaarvoiceNot bizarre boys

Austin-based company founded in 2005

SaaS serving software that collects and displays user generated content, crunches analytics, and extracts data.

Engineering offices in Austin, NYC, London, and San Francisco

Basic stats:Thousands of clients

Hundreds of millions of pieces of content

Hundreds of millions of unique visitors per month

Tens of billions of page-views per month

Edge Traffic

EC2, S3, VPC,Regions, Autoscale,

CloudFormation, ELB...?

Does this mean anything to you?

VPC & SubnetsVPC allows us to choose our internal IP space.

Default Route for All Subnets to IGW● Let's call these subnets all "Public"● Requires all instances to have EIPs before

talking to the internet● EIPs are a limited resource

Public: Default route via IGW

Private: Default route via instance(s) in Public Subnets

Advantage: Most instances in the private subnet can talk to the internet without dealing with an EIP.

Security Groups

Autoscaling

Elastic Load Balancing

● Only Round Robin and Sticky Sessions● Supports HTTP Response code or basic TCP connection Health Checks

{ "AWSTemplateFormatVersion" : "2010-09-09", "Description" : "A text description for the template usage", "Parameters": { // A set of inputs used to customize the template per deployment }, "Mappings": { // Mappings match a key to a corresponding set of named values }, "Resources" : { // The set of AWS resources and relationships between them }, "Outputs" : { // A set of values to be made visible to the stack creator }}

CloudFormation Instance Example

{ "AWSTemplateFormatVersion" : "2010-09-09", "Description" : "Create an EC2 instance running the Amazon Linux 32 bit AMI.”, "Parameters" : { "KeyPair" : { "Description" : "The EC2 Key Pair to allow SSH access to the instance", "Type" : "String" } }, "Resources" : { "Ec2Instance" : { "Type" : "AWS::EC2::Instance", "Properties" : { "KeyName" : { "Ref" : "KeyPair" }, "ImageId" : "ami-75g0061f", "InstanceType" : "m1.small" } } }, "Outputs" : { "InstanceId" : { "Description" : "The InstanceId of the newly created EC2 instance", "Value" : { "Ref" : "Ec2Instance" } } }}

IAM and Console AccessSign-on Credentials

● IAM Console login○ Username, Password, and MFA Time token

Access Credentials● AWS has three API types: REST, Query, & SOAP. ● Each API uses one or more Access Credentials

○ Access Keys for REST and Query APIs○ x.509 Certificates for SOAP API○ EC2 KeyPairs for instance SSH authentication

In the beginning...

A Java application server + a MySQL Database

Scaled by adding in another application server.

Then we just duplicated this entire stack, giving us two "clusters".

Scaled more by adding more and more clusters.

Add in AWS

Decentralization --> Everyone goes fast!

Goals

Full control over AWS resourcesEC2 resources, Autoscale, ELB, S3, etc.

Team IsolationResources created by one team can only be modified/terminated by

that team

3rd Party SolutionenStratus, RightScale, asgard, etc

Good

● enStratus & RightScale provide cloud-agnostic

tools

Bad

● No AWS API Access

● No AWS CLI Tools & SDKs

● Locked into only supported services

Multiple Accounts

Good

● Provides for full resource control with direct API

access

● Protects teams from one another

● Allows for easy accounting on a per-team basis

● May make it easier for external auditors to

determine which teams have

"production" access

Bad

● Inter-team network communications can become

very complicated, relying

on VPN between VPCs ->

Reduced Reliability

● Management of networking is a possible bottleneck

● Shared resources may need to be redundantly built in

every VPC: LDAP, DNS,

Monitoring

Single Shared Account

Good

● Sharing of resources will be simple - just open

access via security groups

between teams

● Reliable networking between teams without need for VPN

● Possibly better performance due to fewer hops

● Certain resources can be shared: LDAP, DNS,

Monitoring, etc.

Bad

● No built-in protections between teams, even with

IAM

● Creates a centralized resource that someone has

to maintain

● Requires us to build tools to use long-term

Nexus, circa August 2012

In more detail...

Nexus is:

● AWS Infrastructure designed with best practices:

○ secure○ highly available○ multi-region○ repeatable

● Cloud building blocks and recipes for all of Engineering

● A Single Account Solution

Philosophy: Engineering teams at Bazaarvoice are free to choose

their own stack, but we want to make Nexus so compelling that it

is the default choice.

(some) Batteries Included

Included:● Bastion Hosts

● NAT Instances

● VPN Connectivity between Regions

● Internal DNS

● Monitoring*

● Centralized Logging*

● Services Discovery*

● Scripts & CloudFormation in Github to create ephemeral

VPCs that look like a

Managed Environment

Dev teams provide

anything required to

run their app, which

probably means:

● Puppet/Chef/etc

● Your actual app

● Deployment process

Nexus Regions

bas·tion (*/ˈbas-chən/*)

NAT Instances

Autoscaling with CloudFormation

Internal DNS

External DNS (Route53)

Records in the .nexus.bazaarvoice.com zone are for you to use.

Badger

Cabertoss

Conformity -> Measurable Efficiency

Limitations & Risks

● Danger! Single Shared Account○ You can wipe out all of a region with a

bad script.

● Single NAT per AZ○ Someone else downloading lots of data from

the internet will affect all other instances sharing your private subnet.

● Single VPN Instance per VPN Destination○ Similar to NAT problem, but worse. ○ Avoid VPN when possible○ If not possible, make your VPN dependency

resilient to lack of bandwidth and network blips

Nexus is a catalyst:

old and busted

waterfall

centralized development

8-10 week release cycle

monolithic app

mysql

solr

java

dev + ops

new and shiny

agile

distributed teams

release anytime

services oriented architecture

cassandra

elasticsearch

whatever

devops

Email: victor@bazaarvoice.comTwitter: @victortrac

Thanks!