better, faster, cheaper infrastructure: apache cloudstack and riak cs

BETTER, FASTER, CHEAPER INFRASTRUCTURE: APACHE

CLOUDSTACK + RIAK CSJohn Burwell

([email protected] | @john_burwell)

Friday, November 8, 13

mailto:[email protected]


http://twitter.com/john_burwell


Who Am I

• Consulting Engineer @ Basho Technologies

• Member, Apache CloudStack PMC

• Ran operations and designed automated provisioning for hybrid analytic/virtualization clouds

• Led architectural design and server-side development of a SaaS physical security platform


http://basho.com

http://basho.com

http://cloudstack.apache.org


... but I submitted a ticket.


Shifting Role Of IT

From Cost Center To Business Opportunity


Shifting Role Of IT

From Cost Center To Business Opportunity

Infrastructure Be Boring


Better

• Resilience

• Scalability

• Service


Faster

• Resource Provisioning

• Systems

• Failure recovery


Cheaper

• Hardware

• Labor

• Hosting


Composable Infrastructure

• Resource Pooling

• Rapid Elasticity

• Self Service

• Metering/Charge Back


Composable Infrastructure

• Resource Pooling

• Rapid Elasticity

• Self Service

• Metering/Charge Back

Late-binding of hardware roles


Got It ... Why Go Private?

• Leverage existing infrastructure investment

• Customization and Control

• Regulatory/Security Requirements

• Cost


Doesn’t vSphere/XCP/XenServer already do that?


Hopefully not ...


Traditional Workload

Management

Network

HypervisorCluster

HypervisorCluster

Shared�� Storage

• Reliable

• Fault tolerant

• Specialized hardware

• IT managed provisioning

• Static scale out model


Traditional Workload

Management

Network

HypervisorCluster

HypervisorCluster


• Reliable

• Fault tolerant

• Specialized hardware

• IT managed provisioning

• Static scale out model

Focused on resource consolidation not agility


Software Is Eating Hardware

• Moore’s Law has pulled fault tolerance up the stack

• Commodity Hardware

• Reduced cost

• Services more workload types

• Dynamic resource allocation


Software Is Eating Hardware

• Moore’s Law has pulled fault tolerance up the stack


• Reduced cost

• Services more workload types

• Dynamic resource allocation

Specialized Hardware Reduces Agility


Cloud Workload• Reliable

• Fault Tolerant


• Self-Service provisioning

• Control Plane -> Dynamic scale out model

SDN

Rack Rack

Object�� Storage

App App


Cloud Workload• Reliable

• Fault Tolerant


• Self-Service provisioning

• Control Plane -> Dynamic scale out model

SDN

Rack Rack

Object�� Storage

App App

Resource optimization increases agility


TraditionalWorkload

CloudWorkload

Reality


Complete Private Cloud

+

Compute Services Object Storage Services


What Is Apache CloudStack?

§Open source software that deploys and manages large networks of virtual machines, as a highly available, highly scalable Infrastructure as a Service (IaaS) cloud computing platform.

§CloudStack is a Top-Level Project at the Apache Software Foundation.


~300: Acknowledged production implementations

~43,000: Hosts in a single CloudStack instance

~85: Project committers


Apache CloudStack Users


Anatomy Of CloudStack• Management Server Farm

• Management and provisioning tasks

• Zone

• Collection of pods, network offerings and secondary storage

• Pod

• Collection of clusters in the same failure boundary

• Cluster

• A grouping of hosts and their associated storage

• Hosts

• Servers onto which services will be provisioned

• Primary Storage

• VM disk storage

• Network

• Logical network associated with service offerings

• Secondary Storage

• Template, snapshot and ISO storage

Zone

CloudStack Pod

Cluster

Host

Host

Network

PrimaryStorage

VM

VM

CloudStack Pod

Cluster

SecondaryStorage


Management Server

• Orchestrates infrastructure management

• A single management server instance can manage 10,000 hosts

• For production, run n +1 management servers with at least one instance in each zone

• Cluster MySQL


Control And Data Planes

• Automation control of resources (e.g. hypervisors, storage, switches, etc)

• Small, linear bandwidth consumption

• Low latency

• Transfer of data between infrastructure components (e.g. templates, snapshots, etc)

• Large, variable bandwidth consumption

• High throughput


Control And Data Plane Scalability

• Separate the control and data plane compute and network traffic

• Data transfer operations offloaded to system virtual machines

• Separate management and storage networks


Primary Vs Secondary Storage

• Random access read/write

• Changes often

• Zone Scoped

• Focused on throughput

• Write once/read many

• Append only

• Zone or Region Scoped

• Focused on reliability


Primary Vs Secondary Storage

• Random access read/write

• Changes often

• Zone Scoped

• Focused on throughput

• Write once/read many

• Append only

• Zone or Region Scoped

• Focused on reliability

Secondary Storage is the lifeboat.


What Is Riak CS?• Simple, reliable storage of large objects accessed using the S3 or Swift

API

• Objects are stored in buckets with a unique key

• Multi-tenant security model with per object ACLs

• Usage Accounting and Reporting

• Open Source (Apache2 Licensed)


What Is Riak CS?• Simple, reliable storage of large objects accessed using the S3 or Swift

API

• Objects are stored in buckets with a unique key

• Multi-tenant security model with per object ACLs

• Usage Accounting and Reporting

• Open Source (Apache2 Licensed)

Your Own Private S3


Anatomy Of Riak CS

• S3/Swift API Proxy + Riak key-value store

• Peer-to-Peer Design: Any node can service any read or accept any write request

• Maintains multiple replicas (default 3) spread across multiple nodes

• Ops Friendly


Anatomy Of Riak CS

• S3/Swift API Proxy + Riak key-value store

• Peer-to-Peer Design: Any node can service any read or accept any write request

• Maintains multiple replicas (default 3) spread across multiple nodes

• Ops Friendly

Horizontally scaleable storage on commodity hardware


Riak CS Use Cases

• Secondary Storage for VM snapshot and template data

• Database Backup Storage

• Serving Static Content

• Application Log Storage


Achieving Commodity Reliability

• Create a zone per power source/backbone network connection

• Establish independent primary storage in each zone

• Utilize intra-datacenter block-level replication between zones

• Distribute Riak CS nodes evenly across zones


The Complete Private Cloud

Management

Network

HypervisorCluster

HypervisorCluster


SDN

Rack Rack

Riak�� CS

App App

Traditional�� Cluster Cloud�� Workload


Up And Running With Riak CS

• Learn more @ http://basho.com/riak-cloud-storage

• Grab it @ http://docs.basho.com/riakcs/latest/riakcs-downloads/

• Follow us @basho

• Help from the [email protected] mailing list or on FreeNode @ #riak

• Contribute @ http://github.com/basho/riak_cs


http://basho.com/riak-cloud-storage

http://basho.com/riak-cloud-storage

http://docs.basho.com/riakcs/latest/riakcs-downloads/




http://twitter.com/basho

http://twitter.com/basho



https://github.com/basho/riak_cs

https://github.com/basho/riak_cs

Join Our Community

• Learn more @ http://cloudstack.apache.org

• Grab it @ http://cloudstack.apache.org/downloads.html

• Follow us @cloudstack

• Help from the [email protected] mailing list or on Freenode @ #cloudstack

• Contribute on the [email protected] mailing list or on Freenode @ #cloudstack-dev




http://cloudstack.apache.org/downloads.html

http://cloudstack.apache.org/downloads.html





Thoughts? Questions?


Thank you!

Deck http://slideshare.net/JohnBurwell1Follow @ john_burwell

Email [email protected]


http://speakerdeck.com/jburwell

http://speakerdeck.com/jburwell





better, faster, cheaper infrastructure: apache cloudstack and riak cs

Technology

apache cloudstack usersfriday

single cloudstack instance

scalable infrastructure

apache software foundation

object storage services

infrastructure components

cheaper infrastructure

storage networksfriday