globo .comOrchestrating Shared Networks, Physical LB and DNS on Cloudstack

Team

Team

Broadcast TV Books and Magazines Music Movies

Pay TV Newspsaper Music Internet

Globo.com‣ 2 Datacenters ‣ Rio de Janeiro (Main Datacenter) ‣ São Paulo (CDN)

‣ More than 1 Tbps of internet connectivity ‣ More than 2000 Servers ‣ More than 400 Developers

New DataCenter in Rio (2015-03)‣ Capacity for more than 10000 servers

Globo.com‣ We are a technology company ‣ Developers and Services providers for :

‣ Content Management System (CMS) ‣ Internet Infra Structure ‣ Connectivity ‣ Hosting ‣ CDN ‣ Information Security ‣ New Technologies (Private Cloud and PaaS)

IaaS at Globo.com‣ CloudStack ‣ Private Cloud ‣ Launched in Jan/2014 ‣ opensource.globo.com

Setup of ACS‣ Advanced Zone Type ‣ Shared Networks ‣ XenServer 6.2 ‣ ACS Version 4.3+

DNS Integration

DNS on unmodified ACS

News APP Network

VR's name resolution is only within network

MySQL Network

VR's name resolution is only within network

News MySQL

Virtual Router(DHCP, DNS)

Virtual Router(DHCP, DNS)

Internal DNS

delegate delegate

Who Is news.mysql.globo.com?

News App

GloboDNS‣ Manage Bind Server ‣ Fork of PowerDNS on Rails ‣ HTTP(S) API ‣ Import data from Bind ‣ Open Source ‣ https://github.com/globocom/GloboDNS

GloboDNS plugin

GloboDNS plugin

GloboDNS plugin

Now the name "news.mysql.globo.com" is globally available to all networks

With GloboDNS plugin

News APP NetworkMySQL Network

news.mysql.globo.com

Virtual Router(DHCP, DNS)

Virtual Router(DHCP, DNS)

newsapp.globo.com

Internal DNS

Naming resolution Naming resolution

XX

Creating a new domain is easy using templates

Administrators are responsible for managing complexity

Load Balancer

LB on unmodified ACS

LB on unmodified ACS

LB on unmodified ACS

LB on unmodified ACS

LB with GloboNetworkAPI‣ LB IP in different network ‣ Doesn't work as LB in ACS ‣ Uses new commands

‣ Hardware-based ‣ GloboNetworkAPI abstract hardware ‣ Usually LB have public IPs, not VMs

LB with GloboNetworkAPI

LB with GloboNetworkAPI

LB with GloboNetworkAPI

LB with GloboNetworkAPI

LB with GloboNetworkAPI

Shared Networks

Shared Networks

Shared NetworksUnmodified ACS

Shared NetworksUnmodified ACS With GloboNetwork

Shared Networks

GloboNetworkAPI

What is GloboNetworkAPI?‣ Open Source REST API for IP Networking Control ‣ Automates provisioning of network resources ‣ Vendor agnostic ‣ Plugin based

Features‣ Cabling documentation (patch-panels/DIO’s capability) ‣ Layer 2 documentation (vlan) ‣ Layer 3 documentation (IP networks) ‣ Automatic allocation of Vlans, Networks and IP’s ‣ IPv6 support ‣ ACL (access control list) documentation/versioning/applying ‣ Load-Balancer support ‣ Automated equipment configuration (plugin based) ‣ User/Equipment Group based authorization

Globo NetworkAPI‣ globonetworkapi.rtfd.org ‣ globonetworkapi-webui.rtfd.org ‣ globonetworkapi-client-python.rtfd.org ‣ https://github.com/globocom/GloboNetworkAPI-

client-java

Environment

Globo NetworkAPI‣ Environment: ‣ Vlan/IP network usable range ‣ Templates for equipment configuration (plugins)

‣ Resources allocated/deployed on demand

Environment Config

Environment Config

Tradicional DC Infrastructure

GloboNetworkAPI + ACS‣ Zones per type of traffic ‣ Type of traffic defines the Environments ‣ ACS maps zones to environments ‣ Xen Servers with tagged vlans (802.1Q)

Virtualized DC Infrastructure

Globo NetworkAPI Load Balancer‣ Load balancer has its own environment ‣ In front of VM networks ‣ LB across zones ‣ LB as an additional plugin on Cloudstack

GloboNetworkAPI WebUI

Roadmap‣ On ACS ‣ Integrating with baseline (GloboDNS on 4.5?) ‣ OAuth2 (in progress) ‣ ACL API ‣ IPv6 Support ‣ Native Load Balancer Support

References‣ opensource.globo.com ‣ https://github.com/globocom/GloboDNS ‣ https://github.com/globocom/cloudstack ‣ globonetworkapi.rtfd.org ‣ https://cwiki.apache.org/confluence/display/CLOUDSTACK/Bind

+and+PowerDNS+integration+by+Globo+DNSAPI

We are hiring !

https://github.com/globocom/IWantToWorkAtGloboCom

Questions? iaas@corp.globo.com

Maurício Kilikrates A N dos Anjos <mkilikrates@corp.globo.com>

Marcus Vinicius Gonçalves Cesário <marcus.vinicius@corp.globo.com>Silvano Nogueira Buback <silvano@corp.globo.com>