cloud expo east 2013: essential open source software for building the open cloud
DESCRIPTION
Cloud computing is more than a buzz-phrase it’s a transformative IT paradigm shift. The emphasis in the cloud is on elasticity, scalability, agility and open. Not just open standards but open APIs and open source. The delivery of software is also going through a paradigm shift. Open source software was often a commoditization of a market leader; Unix to Linux or Oracle to MySQL what’s changing is that the iterative nature, user context and the motto of releasing early and often are driving real innovation in open source. This session will cover those essential open source technologies for delivering cloud computing in the enterprise. Speaker Bio: Mark Hinkle is the Senior Director, Open Source Solutions at Citrix Systems Inc. He joined Citrix as a result of their July 2011 acquisition of Cloud.com where he was their Vice President of Community. He is currently responsible for Citrix open source efforts around the open source cloud computing platform, Apache CloudStack and the Xen Hypervisor. Previously he was the VP of Community at Zenoss Inc., a producer of the open source application, server, and network management software, where he grew the Zenoss Core project to over 100,000 users and 20,000 organizations on all seven continents. He also is a longtime open source expert and author having served as Editor-in-Chief for both LinuxWorld Magazine and Enterprise Open Source Magazine. His blog on open source, technology, and new media can be found at http://www.socializedsoftware.com.TRANSCRIPT
Essential Open Source Software for Building the Open Cloud
Mark R. Hinkle Sr. Director , OPEN SOURCE SOLUTIONS
Citrix Systems INC.
@mrhinkle
Cloud Computing Expo East 2013
Mark Hinkle, Sr. Director, Open Source Solutions
• Dedicated to the success of the Apache CloudStack, Open Daylight & Xen Project Communi3es on Citrix behalf
• Run BuildACloud.org learning ac3vi3es all over the world • Joined Citrix via Cloud.com acquisi3on July 2011 • Zenoss Core Open Source project to 100,000 users, 1.5
million downloads • Former LinuxWorld Magazine Editor-‐in-‐Chief • Open Management ConsorGum organizer • Author -‐ “Windows to Linux Business Desktop MigraGon” –
Thomson • NetDirector Project -‐ Open Source Configura3on
Management • Some3mes Author and Blogger at SocializedSoJware.com • NetworkWorld Open Source Subnet
Essential OSS for Building the Open Cloud by @mrhinkle
2
Why Open Source and the Cloud Computing?
• User-‐Driven Context from Solving Real Problems • Lower Barrier to Par3cipa3on • Larger user base, users helping users • Aggressive release cycles stay current with the state-‐of-‐the-‐art
• Open Source innova3ng faster than commercial • Open data, Open standards, Open APIs
Essential OSS for Building the Open Cloud by @mrhinkle
3
Quick Cloud Computing Overview or the Obligatory “What is the Cloud Explanation”
Five Characteristics of Cloud
1. On-‐Demand Self-‐Service
2. Broad Network Access 3. Resource Pooling 4. Rapid Elas3city 5. Measured Service
Essential OSS for Building the Open Cloud by @mrhinkle
5
Cloud Computing Service Models
USER CLOUD a.k.a. SOFTWARE AS A SERVICE Single application, multi-tenancy, network-based, one-to-many delivery of applications, all users have same access to features. Examples: Salesforce.com, Google Docs, Red Hat Network/RHEL DEVELOPMENT CLOUD a.k.a. PLATFORM-AS-A-SERVICE Application developer model, Application deployed to an elastic service that autoscales, low administrative overhead. No concept of virtual machines or operating system. Code it and deploy it. Examples: VMware CloudFoundry, Google AppEngine, Windows Azure, Rackspace Sites, Red Hat OpenShift, Active State Stackato, Appfog SYSTEMS CLOUD a.k.a INFRASTRUCTURE-AS-A-SERVICE Servers and storage are made available in a scalable way over a network. Examples: EC2,Rackspace CloudFiles, OpenStack, CloudStack, Eucalyptus, OpenNebula
Essential OSS for Building the Open Cloud by @mrhinkle
6
Deployment Models: Public, Private & Hybrid
Essential OSS for Building the Open Cloud by @mrhinkle
7
Building Open Source Clouds
First Question…What do you want to do?
ü Self-‐Service enterprise workloads ü Customer facing Applica3ons ü Dev/Test environments ü Provide Something-‐as-‐a-‐Service ü Deploy similar workloads at scale ü All of the Above
9
Essential OSS for Building the Open Cloud by @mrhinkle
Generic Cloud Architecture
Essential OSS for Building the Open Cloud by @mrhinkle
10
Building Open Source Clouds
Virtualized Compute
Virtualized Storage
Virtualized Networking
Infrastructure Orchestra3on
Pla^orm-‐as-‐a-‐Service
Infrastructure-‐as-‐a-‐Service
Ops Management Tools
Enterprise 3rd Party Developers
Essential OSS for Building the Open Cloud by @mrhinkle
Config Mgmt Monitoring
Provisioning Orchestra3on
Choose what you want, customize for needs.
Essential OSS for Building the Open Cloud by @mrhinkle
The Three Virtual Pillars of Cloud Computing
13
Compu
te
Network
Storage
Hypervisors
Open Source • Xen, Project Xen Cloud PlaMorm (XCP) • KVM – Kernel-‐based VirtualizaGon • VirtualBox* -‐ Oracle supported Virtualiza3on Solu3ons • OpenVZ* -‐ Container-‐based, Similar to Solaris Containers or BSD Zones • LXC – User Space chrooted installs Proprietary • VMware • Citrix Xenserver (based • Microsoc Hyper-‐V • OracleVM (Based on OS Xen)
Essential OSS for Building the Open Cloud by @mrhinkle
14
Scale-Up or Scale-Out
VerGcal Scaling (Scale-‐Up) Allocate addi3onal resources to VMs, requires a reboot, no need for distributed app logic, single-‐point of OS failure
Horizontal Scaling (Scale-‐Out) Applica3on needs logic to work in distributed fashion (e.g. HA-‐Proxy and Apache, Hadoop)
Essential OSS for Building the Open Cloud by @mrhinkle
15
Sourcing Cloud Appliances Tool/Project What you can do with them
Bitnami BitNami provides free, ready to run environments for your favorite open source web applica3ons and frameworks, including Drupal, Joomla!, Wordpress, PHP, Rails, Django and many more.
Boxgrinder BoxGrinder is a set of projects that help you grind out appliances for mul3ple virtualiza3on and Cloud providers
Oz Command-‐line tool that has the ability to create images for common Linux distribu3ons to run on KVM
SUSE Studio SUSE Studio supports building and deploying directly to cloud services such as Amazon EC2.
Essential OSS for Building the Open Cloud by @mrhinkle
16
Cloud Computing Storage Project DescripGon
Ceph Distributed file storage system developed by DreamHost
GlusterFS Scale Out NAS system aggrega3ng storage over Ethernet or Infiniband
OpenStack Storage
Long-‐term object storage system
Riak CS Riak CS is open source socware designed to provide simple, available, distributed cloud storage at any scale. Riak CS is S3-‐API compa3ble and supports per-‐tenant repor3ng for billing and metering use cases.
Sheepdog Distributed storage for KVM hypervisors
Essential OSS for Building the Open Cloud by @mrhinkle
17
Software Defined Networking (SDN)
Cloud Promise, Reality and Networks
Cloud Promise Cloud Reality Centralized ConfiguraGon and AutomaGon
Without true virtualiza3on, network devices must s3ll be manually configured.
Instant Self-‐Service Provisioning
In a physical network, it could take a long 3me for network engineer to provision new services.
ElasGcity and Scalability By horizontally scaling up the physical network, elas3city is lost.
Designed for Failure Failover can be automated and physical network limita3ons can be alleviated.
Source: Midokura
Essential OSS for Building the Open Cloud by @mrhinkle
19
Open Flow
OpenFlow enables networks to evolve, by giving a remote controller the power to modify the behavior of network devices, through a well-‐defined "forwarding instruc3on set". The growing OpenFlow ecosystem now includes routers, switches, virtual switches, and access points from a range of vendors.
Image from http://www.open!ow.org/documents/open!ow-wp-latest.pdf
Essential OSS for Building the Open Cloud by @mrhinkle
20
Overview of Software Defined Networking
Business Applica3ons
Network Services
SDN Control Software
API API
Network Devices Network Devices Network Devices
Network Devices Network Devices Network Devices
Application Layer
Control Layer
Infrastructure Layer
Control Data Plane Interface (e.g. OpenFlow)
Essential OSS for Building the Open Cloud by @mrhinkle
21
Software Defined Networking (SDN) Project Description
Floodlight The Floodlight controller is an enterprise-‐class, Apache-‐licensed, Java-‐based OpenFlow Controller.
Indigo Indigo is an open source project to support OpenFlow on a range of physical switches. By leveraging hardware features of Ethernet switch ASICs, Indigo supports high rates for high port counts, up to 48 10-‐gigabit ports. Mul3ple gigabit pla^orms with 10-‐gigabit uplinks are also supported.
Open Daylight Linux Founda3on Collabora3ve Project based on Cisco One Controller and plugins from numerous vendors in development. E.g IBM DOVE
OpenStack “Quantum” Networking
Pluggable, scalable, API-‐driven network and IP management
Open vSwitch Open vSwitch is a open source (ASL 2.0), mul3layer virtual switch designed to enable massive network automa3on through programma3c extension, while s3ll suppor3ng standard management interfaces and protocols (e.g. NetFlow, sFlow, SPAN, RSPAN, CLI, LACP, 802.1ag).
Essential OSS for Building the Open Cloud by @mrhinkle
22
Cloud Orchestration
Compute Clouds (IaaS)
Year Started License VirtualizaGon Technologies
Apache CloudStack
2008 Apache Xenserver, Xen Cloud Pla^orm, KVM, VMware (Hyper-‐V developing)
Eucalyptus 2006 GPL Xen, KVM, VMware (commercial version)
OpenNebula 2005 Apache Xen, KVM, VMware
OpenStack 2010 (Developed by NASA by Anso Labs previously)
Apache VMware ESX and ESXi, , Xen, Xen Cloud Pla^orm KVM, LXC, QEMU and Virtual Box
Numerous companies are building cloud software on OpenStack including Nebula, Piston Inc., CloudScaling
Essential OSS for Building the Open Cloud by @mrhinkle
24
OpenStack – Ecosystem of Projects
Enterprise Message Queue based on Rabbit MQ (ESB)
Object Storage “Swic”
Image Service “Glance
”
Compute “Nova”
Dashboard “Horizon”
KVM, VMware, Xen Cloud Pla^orm Ceph, Gluster
Advanced Cloud and Networking services accessing the Quantum API
Firewall Service
Gateway Service
Quantum
Netw
orking Fabric REST API Plugins
OpenvSwitch Quantum Plugin-‐ins Id
en3ty Services “Ke
ystone
” API
20+ Collective projects hosted at: https://launchpad.net/openstack
Essential OSS for Building the Open Cloud by @mrhinkle
25
Platform-as-a-Service (PaaS) Project Year Started Sponsors Languages/Frameworks
CloudFoundry 2011 VMware Spring for Java, Ruby for Rails and Sinatra, node.js, Grails, Scala on Lic and more via partners (e.g. Python, PHP)
Cloudify 2012 Gigaspaces [Groovy for deployment recipes]
OpenShic 2011 Red Hat Java, Ruby, PHP, Perl and Python
Stackato 2012 Ac3veState Java, Python, PHP, Ruby, Perl, Node.js, others
WSO2 Stratus 2010 WSO2 Jboss, Java EE6
Essential OSS for Building the Open Cloud by @mrhinkle
26
Cloud APIs
• jclouds • libcloud • deltacloud • fog
Essential OSS for Building the Open Cloud by @mrhinkle
27
Cloud Management Tools
Automation in the Cloud
Meat Cloud Cloud Operations
Essential OSS for Building the Open Cloud by @mrhinkle
29
4 Types of Management Tools
Provisioning Installation of operating systems and other software Configuration Management Sets the parameters for servers, can specify installation parameters Orchestration/Automation Automate tasks across systems Monitoring Records errors and health of IT infrastructure
Essential OSS for Building the Open Cloud by @mrhinkle
30
Management Toolchains
Configura3on
Patching and
Provisioning
Monitoring
Toolchain (n): A set of tools where the output of one tool becomes the input of another tool
Essential OSS for Building the Open Cloud by @mrhinkle
31
Provisioning Project Installation Targets
Apache Provisionr(incuba3ng)
Can provision 10s to 1000s of machines on various clouds.
Cobbler Distributed virtual infrastructure using koan (kickstart of a network to PXE boot VMs) for Red Hat, OpenSUSE Fedora, Debian, Ubuntu VMs
Crowbar (Bare metal provisioning)
JuJu Public Clouds -‐ Amazon Web Services HP Cloud, Private OpenStack clouds, Bare Metal via MAAS.
Salt Cloud Tool to provision “salted” VMs that can then be updated by a central server via ZeroMQ
Essential OSS for Building the Open Cloud by @mrhinkle
32
Configuration Management Tools
Project Year Started Language License Client/Server
Cfengine 1993 C Apache Yes
Chef 2009 Ruby Apache Chef Solo – No Chef Server -‐ Yes
Puppet 2004 Ruby GPL Yes & standalone
Salt 2011 Python Apache yes
Essential OSS for Building the Open Cloud by @mrhinkle
33
Automation/Orchestration Tools Project DescripGon
Ansible Ansible's SSH-‐key based access allows contributors to the Fedora Project to assist in automa3ng infrastructure while having access limited appropriately.
Capistrano U3lity and framework for execu3ng commands in parallel on mul3ple remote machines, via SSH. It uses a simple DSL that allows you to define tasks, which may be applied to machines in certain roles
RunDeck Rundeck is an open-‐source process automa3on and command orchestra3on tool with a web console.
Func Func provides a two-‐way authen3cated system for generically execu3ng tasks, integra3ons with puppet and cobbler.
MCollec3ve The Marionese Collec3ve AKA MCollec3ve is a framework to build server orchestra3on or parallel job execu3on systems.
Salt Execute arbitrary shell commands or choose from dozens of pre-‐built modules of common (or complex) commands.
Scalr Provide scaling across mul3ple cloud compu3ng pla^orms, integrates with Chef.
Essential OSS for Building the Open Cloud by @mrhinkle
34
Configuration Management Tools
Project Year Started Language License Client/Server
Cfengine 1993 C Apache Yes
Chef 2009 Ruby Apache Chef Solo – No Chef Server -‐ Yes
Puppet 2004 Ruby GPL Yes & standalone
Salt 2011 Python Apache yes
Essential OSS for Building the Open Cloud by @mrhinkle
35
Automation/Orchestration Tools Project DescripGon
Ansible Ansible's SSH-‐key based access allows contributors to the Fedora Project to assist in automa3ng infrastructure while having access limited appropriately.
Capistrano U3lity and framework for execu3ng commands in parallel on mul3ple remote machines, via SSH. It uses a simple DSL that allows you to define tasks, which may be applied to machines in certain roles
RunDeck Rundeck is an open-‐source process automa3on and command orchestra3on tool with a web console.
Func Func provides a two-‐way authen3cated system for generically execu3ng tasks, integra3ons with puppet and cobbler.
MCollec3ve The Marionese Collec3ve AKA MCollec3ve is a framework to build server orchestra3on or parallel job execu3on systems.
Salt Execute arbitrary shell commands or choose from dozens of pre-‐built modules of common (or complex) commands.
Scalr Provide scaling across mul3ple cloud compu3ng pla^orms, integrates with Chef.
Essential OSS for Building the Open Cloud by @mrhinkle
36
Monitoring Tools
License Type of Monitoring CollecGon Methods
Cac3 / RRDTool GPL Performance SNMP, syslog
Graphite Apache 2.0 Performance Agent
Nagios GPL Availability SNMP,TCP, ICMP, IPMI, syslog
Zabbix GPL Availability/ Performance and more
SNMP, TCP/ICMP, IPMI, Synthe3c Transac3ons
Zenoss GPL Availability, Performance, Event Management
SNMP, ICMP, SSH, syslog, WMI
Essential OSS for Building the Open Cloud by @mrhinkle
37
Conceptual Automated Toolchain
BootStrapped Image CloudStack OpenStack
ConfiguraGon Puppet Chef
Start/Stop Services RunDeck Capistrano MCollec3ve
Provision Cobbler
SUSE Stuido
Monitoring Nagios Zenoss Cac3
Generate Images SUSE Studio BoxGrinder
Essential OSS for Building the Open Cloud by @mrhinkle
38
Questions?
Essential OSS for Building the Open Cloud by @mrhinkle
39
Slides Can be Viewed and Downloaded at: hsp://www.slideshare.net/socializedsocware/
Copyright Mark R. Hinkle, available under the CCbySA license some rights reserved. 2012 -2013
Professional: [email protected] Personal: [email protected]
Phone: 919.228.8049
Personal: http://www.socializedsoftware.com
Twitter: @mrhinkle
Mark R. Hinkle Senior Director, Open Source Solutions Citrix Systems Inc. Open Source Enthusiast