Fossetcon: Crash Course on Open Source Cloud Computing

Download Fossetcon: Crash Course on Open Source Cloud Computing

Post on 22-Jan-2015

503 views

Category:

Software

4 download

Embed Size (px)

DESCRIPTION

This crash course is designed to give an overview of cloud computing architecture and the open source software that can be used to deploy and manage a cloud computing environment. Topics to be discussed in this session will include virtualization (KVM, LXC, and Xen Project), orchestration (Apache CloudStack, Eucalyptus, Open Nebula, and OpenStack), and storage (GlusterFS, Ceph, and others). The talk will also provide insight into how to deliver Platform-as-a-Service (PaaS) and what technologies can be used to compliment this evolving cloud computing paradigm. Systems administrators and IT generalists will leave the discussion with a general overview of the options at their disposal to effectively build and manage their own cloud computing environments using free and open source software and understand the capabilities and benefits of a host of technologies. [Updated with new Docker projects]

TRANSCRIPT

<ul><li> 1. Crash Course In Open SourceCloud ComputingMark HinkleSenior Director, Open Source SolutionsCitrix Inc.mark.hinkle@citrix.commrhinkle@gmail.com@mrhinkleLast updated: 9/11/2014</li></ul><p> 2. By Mark R. Hinkle@mrhinklemrhinkle@gmail.comABOUT MEI Help Build Open Source EcosystemsOpen Source Experience Manage Citrix Open Source Business Office Apache CloudStack Committer and PMC Member Advisory boards Gluster and Xen Project Joined Citrix via Cloud.com acquisition July 2011 Zenoss Core open source project to 100,000 users,1.5 million downloads Former LinuxWorld Magazine Editor-in-Chief Open Management Consortium organizer Author - Windows to Linux Business DesktopMigration Thomson NetDirector Project - Open Source ConfigurationManagementFOSSETCON 2014 - Crash Course in Open Source Cloud Computing 3. http://www.slideshare.net/socializedsoftwareAttribution You must give appropriate credit, provide a link to the license, and indicate if changeswere made. You may do so in any reasonable manner, but not in any way that suggests the licensorendorses you or your use.ShareAlike If you remix, transform, or build upon the material, you must distribute your contributionsunder the same license as the original.By Mark R. Hinkle@mrhinklemrhinkle@gmail.comSlides Available on Slideshare:Creative Commons Attributions-ShareAlike 4.0 InternationalShare copy and redistribute the material in any medium or formatAdapt remix, transform, and build upon the materialfor any purpose, even commercially.The licensor cannot revoke these freedoms as long as you follow the license terms.FOSSETCON 2014 - Crash Course in Open Source Cloud Computing 4. By Mark R. Hinkle@mrhinklemrhinkle@gmail.comAGENDA Vetting Open Source Cloud Projects What is Cloud in 60 Seconds Virtualization Infrastructure-as-a-Service Platform-as-a-Service SDN Open Source for the Amazon Web ServicesFOSSETCON 2014 - Crash Course in Open Source Cloud Computing 5. Code Velocity Committers Committer Reputation User-driven or Vendor-DrivenInnovation User Activity Corporate Support* Reputation of Foundation*By Mark R. Hinkle@mrhinklemrhinkle@gmail.comVETTING OPEN SOURCEHPowR caOn yJouE telCl if TtheSyre LegitFOSSETCON 2014 - Crash Course in Open Source Cloud Computing 6. http://www.ohloh.net http://activity.openstack.orgBy Mark R. Hinkle@mrhinklemrhinkle@gmail.comOPEN SOURCE ANALYSISVisualizing Community ActivityFOSSETCON 2014 - Crash Course in Open Source Cloud Computing 7. By Mark R. Hinkle@mrhinklemrhinkle@gmail.com60 SECOND CLOUD DEFINITIONJust because Software Marketing Guys Think its the Internet5 CHARACTERISTICS OF CLOUD1. On-Demand Self-Service2. Broad Network Access3. Resource Pooling4. Rapid Elasticity5. Measured ServiceUser Cloud a.k.a.SOFTWARE-AS-A-SERVICEDeveloper Cloud a.k.a.PLATFORM-AS-A-SERVICESystems Cloud a.k.a.INFRASTRUCTURE-AS-A-SERVICEFOSSETCON 2014 - Crash Course in Open Source Cloud Computing 8. By Mark R. Hinkle@mrhinklemrhinkle@gmail.comSCALE-UP SCALE OUTElasticity and the cloudVertical Scaling (Scale-Up)Allocate additional resources toVMs, requires a reboot, no need fordistributed app logic, single-point ofOS failureHorizontal Scaling (Scale-Out)Application needs logic to work indistributed fashion (e.g. HA-Proxyand Apache Hadoop)FOSSETCON 2014 - Crash Course in Open Source Cloud Computing 9. HYPERVISORS AND CONTAINERSDifferences in virtualizationBy Mark R. Hinkle@mrhinklemrhinkle@gmail.comType 1 HypervisorsVMware, Xen Project, Hyper-VType 2 HypervisorsKVM, VirtualBoxFOSSETCON 2014 - Crash Course in Open Source Cloud ComputingContainersLXC 10. By Mark R. Hinkle@mrhinklemrhinkle@gmail.comVIRTUALIZATIONCarving up compute resourcesOPEN SOURCE Xen Project Citrix XenServer KVM VirtualBox OpenVZ LXCPROPRIETARY VMware Microsoft Hyper-V OracleVM (Based on Xen Project)FOSSETCON 2014 - Crash Course in Open Source Cloud Computing 11. OPEN VIRTUALIZATION FORMATSVirtualization PayloadsFormats for hypervisors/cloudtechnologies: Amazon - AMI KVM QCOW2 VMware VMDK Xen Project IMG Hyper-V - VHD Virtual Hard Disk LXC local file system/mount point -By Mark R. Hinkle@mrhinklemrhinkle@gmail.comOpen VirtualizationFormat (OVF) is anopen standard forpackaging anddistributing virtualappliances or moregenerally software tobe run in virtualmachines.Docker*FOSSETCON 2014 - Crash Course in Open Source Cloud Computing 12. Lets your run a Linux system within A container is a group of processes on aLinux box, put together the provide anisolated environment From the inside, it looks like a VM Externally it looks like normal processes chroot on steroidsBy Mark R. Hinkle@mrhinklemrhinkle@gmail.comLINUX CONTAINERS (LXC)Lightweight Linux Virtualizationanother Linux systemFOSSETCON 2014 - Crash Course in Open Source Cloud Computing 13. Different file formats for virtual machines VMware uses vmdk file format, Xen and Hyper-V use VHD, KVM uses Raw or QCOW2 Guest images may be processor architecture VMware and Xen can manage SCSI devices, but KVM and Xen can use virtio drivers but not VMware uses a proprietary agent inside theguest OS (VMware tools) which does not workwith Xen or KVM Xen uses VirtIo and ParaVirtualized drivers, XenBy Mark R. Hinkle@mrhinklemrhinkle@gmail.comTHE PORTABILITY PROBLEMContainers compared to Hardware VirtualizationboundKVM cannotVMwareusesFOSSETCON 2014 - Crash Course in Open Source Cloud Computing 14. Code Application is stored Build Code is built (Jenkins) Test Unit tests areBy Mark R. Hinkle@mrhinklemrhinkle@gmail.comCONTINUOUS INTEGRATIONRebuild Applications on any Cloud and/or Virtualized Infrastructurein a repository(Subversion,Git)automated (Jenkins) Deploy Deploy code toserver various waysCodeBuildTestDeployThoughtworks Go Open SourceContinuous Deliver SystemFOSSETCON 2014 - Crash Course in Open Source Cloud Computing 15. PACKER MULTIPLATFORM VMCREATIONPacker is easy to use and automates thecreation of any type of machine image. Itembraces modern configurationmanagement by encouraging you to useautomated scripts to install and configurethe software within your Packer-madeimages.By Mark R. Hinkle@mrhinklemrhinkle@gmail.comTo learn more please visit:www.packer.ioOpen source Automation for VMsFOSSETCON 2014 - Crash Course in Open Source Cloud Computing 16. DOCKER CONTAINER PACKAGINGOpen source LXC Packaging EngineDocker is an open-source project to easilycreate lightweight, portable, self-sufficientcontainers from any application. The samecontainer that a developer builds and testson a laptop can run at scale, in production,on VMs, bare metal, public clouds andmore.By Mark R. Hinkle@mrhinklemrhinkle@gmail.comTo learn more please visit:www.docker.ioFOSSETCON 2014 - Crash Course in Open Source Cloud Computing 17. Compliment to LXC not a replacement Managed daemonized processes on Linux Create ability to re-use and manage similar Content agnostic Hardware agnostic Easy to automate Integrated with other tools: Chef, OpenShift,By Mark R. Hinkle@mrhinklemrhinkle@gmail.comWHAT IS DOCKERSystem for Managing and Deploying LXC Containersusing LXCapplicationsPuppet, VMware, etc.FOSSETCON 2014 - Crash Course in Open Source Cloud Computing 18. DOCKERS GROWINGECOSYSTEMBy Mark R. Hinkle@mrhinklemrhinkle@gmail.comFOSSETCON 2014 - Crash Course in Open Source Cloud Computing 19. Kubernetes builds on top of Docker toconstruct a clustered container schedulingservice. Kubernetes enables users to aska cluster to run a set of containers. Thesystem will automatically pick workernodes to run those containers on, whichwe think of more as "scheduling" than"orchestrationTo learn more please visit:https://github.com/GoogleCloudPlatform/kubernetes Greek for ShipmasterBy Mark R. Hinkle@mrhinklemrhinkle@gmail.comKUBERNETESContainer Cluster Management SchedulerFOSSETCON 2014 - Crash Course in Open Source Cloud Computing 20. DOCKER RELATEDPROJECTS Fig -Fast, isolated development environments Flynn - Next-generation application platform Panamax Drag-and-Drop Docker Containerization Project Atomic JEOS designed to run Dockercontainers Weave The Docker Network 13,000+ Docker-related repos on GithubBy Mark R. Hinkle@mrhinklemrhinkle@gmail.comFOSSETCON 2014 - Crash Course in Open Source Cloud Computing 21. Apache Mesos is a cluster manager that simplifies thecomplexity of running applications on a shared pool ofservers. Largely supported by Twitter, used by LinkedIn,AirBNB too.Features Fault-tolerant replicated master using ZooKeeper Scalability to 10,000s of nodes Isolation between tasks with Linux Containers Multi-resource scheduling (memory and CPU aware) Java, Python and C++ APIs for developing newBy Mark R. Hinkle@mrhinklemrhinkle@gmail.comAPACHE MESOSOne to many tools for managing large numbers of devicesparallel applications Web UI for viewing cluster stateTo learn more please visit:http://mesos.apache.org/FOSSETCON 2014 - Crash Course in Open Source Cloud Computing 22. ZooKeeper is a centralized service formaintaining configuration information,naming, providing distributedsynchronization, and providing groupservices. All of these kinds of servicesare used in some form or another bydistributed applicationsBy Mark R. Hinkle@mrhinklemrhinkle@gmail.comAPACHE ZOOKEEPERCentralized Server to Service Distributed AppsTo learn more please visit:http://zookeeper.apache.org/FOSSETCON 2014 - Crash Course in Open Source Cloud Computing22 23. INFRASTRUCTURE-AS-A-SERVICECompute OrchestrationProject Year Started License VirtualizationBy Mark R. Hinkle@mrhinklemrhinkle@gmail.comFOSSETCON 2014 - Crash Course in Open Source Cloud ComputingTechnologiesApacheCloudStack2008 Apache (Bare Metal), Xenserver,KVM, LXC VMware Hyper-VEucalyptus 2006 GPL Xen, KVM, VMware(commercial version)OpenNebula 2005 Apache Xen, KVM, VMwareOpenStack 2010 (Developed byNASA by Anso Labspreviously)Apache VMware ESX and ESXi, ,Xen, XenServer, KVM,LXC, QEMU and VirtualBox 24. By Mark R. Hinkle@mrhinklemrhinkle@gmail.comOPENSTACKThe Boy Band of the Open Source Cloud FOSSETCON 2014 - Crash Course in Open Source Cloud Computing 25. By Mark R. Hinkle@mrhinklemrhinkle@gmail.comOPENSTACK SHAREDSSpaEn CRomVputIe,C StoEragSe and NetworkingIDENTITYSERVICEIMAGESERVICETELEMETRYSERVICEFOSSETCON 2014 - Crash Course in Open Source Cloud ComputingORCHESTRATIONSERVICE 26. EVEN MORE OPENSTACK PROJECTSSpan Compute, Storage and Networking TroveDatabase Service IronicBare Metal (Ironic) MarconiQueue ServiceBy Mark R. Hinkle@mrhinklemrhinkle@gmail.com CinderBlock Storage Service CeilometerMetering/Monitoring HeatOrchestrationFOSSETCON 2014 - Crash Course in Open Source Cloud Computing 27. OPENSTACK SOLUTION PROVIDERSIf you cant do it yourselfOpenStack is not a product. If you are building a large infrastructure, itsmore like a tool kit. It gives you a lot of technologies that do take a lot ofeffort to integrate.Chris Kemp, OpenStack Board Member and Co-FounderBy Mark R. Hinkle@mrhinklemrhinkle@gmail.comFOSSETCON 2014 - Crash Course in Open Source Cloud ComputingCEO of Piston Computing 28. Deltacloud(ruby) Daisein(java) Jclouds(java) Libcloud(python) Fog(ruby)By Mark R. Hinkle@mrhinklemrhinkle@gmail.comCLOUD APISEverything (should) have an API in the CloudFOSSETCON 2014 - Crash Course in Open Source Cloud Computing 29. Project DescriptionCeph Distributed file storage system developed by DreamHost -&gt;GlusterFS Scale Out NAS system aggregating storage over Ethernet orRiak CS Riak CS is open source software designed to provide simple,available, distributed cloud storage at any scale. Riak CS is S3-API compatible and supports per-tenant reporting for billing andmetering use cases. (object)Sheepdog Distributed storage for KVM hypervisors, distributed iSCSIBy Mark R. Hinkle@mrhinklemrhinkle@gmail.comCLOUD STORAGEVirtualized, Distributed usually on Commodity HardwareInkTank -&gt; Red Hat (block, object, file)Infiniband (file)OpenStackStorageLong-term object storage system (object)FOSSETCON 2014 - Crash Course in Open Source Cloud Computing 30. Project Sponsors Languages/FrameworksSpring for Java, Ruby for Rails andSinatra, node.js, Grails, Scala onLift and more via partners (e.g.Python, PHP)Cloudify Gigaspaces [Groovy for deployment recipes]OpenShift Origin Red Hat Java, Ruby, PHP, Perl and PythonApache Stratos WSO2 - &gt;Apache Stratus PHP, Tomcat, MySQL cartridgesBy Mark R. Hinkle@mrhinklemrhinkle@gmail.comPLATFORM-AS-A-SERVICEAbstracted Cloud-Scale Run-Time EnvironmentsCloudFoundry VMware -&gt; Pivotal -&gt; CloudFoundryFoundationFOSSETCON 2014 - Crash Course in Open Source Cloud Computing 31. Decoupling of the control and data planes of the network toimprove efficiency. Communication from a SDN controller via aprotocol to network devices both physical and virtual.Abstractions allow for programmable networks.Network can be changed quickly via a controllerNetwork offerings can match virtualization offerings for finergrained security in a highly volatile compute landscape.By Mark R. Hinkle@mrhinklemrhinkle@gmail.comSOFTWARE DEFINEDVNirtuEalizTatiWon mOeetRs thKe neItwNorkG(SDN)AutomationDynamic NetworksSecurityHeterogeneous ManagementSingle control point for various devices.FOSSETCON 2014 - Crash Course in Open Source Cloud Computing 32. API APINetwork ServicesSDN OVERVIEWControl Data Plane Interface (e.g. OpenFlow)Network Devices Network Devices Network DevicesNetwork Devices Network Devices Network DevicesBy Mark R. Hinkle@mrhinklemrhinkle@gmail.comBusiness ApplicationsSDNControlSoftwareApplicationLayerControlLayerInfrastructureLayerFOSSETCON 2014 - Crash Course in Open Source Cloud Computing 33. BENEFITS OF SDNNetwork Virtualization is the final frontier of Software Defined DatacenterBy Mark R. Hinkle@mrhinklemrhinkle@gmail.com Dynamically update networks Automate networkfunctionality Program security into thenetwork Centrally apply policies tonetwork and services Optimize networksFOSSETCON 2014 - Crash Course in Open Source Cloud Computing 34. OpenFlow enables networks toevolve, by giving a remotecontroller the power to modifythe behavior of networkdevices, through a well-defined"forwarding instruction set".The growing OpenFlowecosystem now includesrouters, switches, virtualswitches, and access pointsfrom a range of vendors.By Mark R. Hinkle@mrhinklemrhinkle@gmail.comOPENFLOWVirtualization meets the networkFOSSETCON 2014 - Crash Course in Open Source Cloud Computing 35. OPEN SOURCE SDNSoftware Defined Network Controllers and moreFloodlight The Floodlight Open SDN Controller is an enterprise-class, Apache-licensed, Java-based OpenFlowController. It is supported by a community of developers including a number of engineers from Big SwitchNetworks. - See more at: http://www.projectfloodlight.org/floodlight/#sthash.9IhA1Ih5.dpufIndigo Indigo is an open source project aimed at enabling support for OpenFlow on physical and hypervisorswitches. Big Switch has helped numerous companies OpenFlow enable their equipment, and weprovide firmware for a number of popular switches. Indigo is the basis of Switch Ligh...</p>