optimizing cloud foundry and openstack for large scale deployments

32
1 OpenStack Summit May 12-16, 2014 Atlanta, Georgia Optimizing OpenStack for large scale Cloud Foundry deployments Animesh Singh, Daniel Krook, Jason Anderson, Kalonji Bankole @animeshsingh @danielkrook @andersonljason @k_bankole

Upload: animesh-singh

Post on 19-Aug-2014

7.763 views

Category:

Engineering


13 download

DESCRIPTION

 

TRANSCRIPT

Page 1: Optimizing Cloud Foundry and OpenStack for large scale deployments

1

OpenStack Summit May 12-16, 2014 Atlanta, Georgia

Optimizing OpenStack for large scale Cloud Foundry deployments

Animesh Singh, Daniel Krook, Jason Anderson, Kalonji Bankole@animeshsingh @danielkrook @andersonljason @k_bankole

Page 2: Optimizing Cloud Foundry and OpenStack for large scale deployments

2 © 2014 IBM Corporation

OAuth

OSLC Infrastructureas a Service

Platformas a Service

Softwareas a Service

API economy

Cloudoperating

environment

Software-defined

environment

TOSCA

http://www.ibm.com/developerworks/cloud/library/cl-open-architecture/

IBM cloud and open technologies

At all tiers, IBM is committed to building its cloud on an open cloud architecture

Products and services built on open source and open standards benefit IBM and customers

Page 3: Optimizing Cloud Foundry and OpenStack for large scale deployments

3 © 2014 IBM Corporation

Gold Sponsors

IBM is #2 in contributions to integrated projects

IBM is working to accelerate OpenStack Foundation success…

Because an open interoperable Cloud is critical for flexible cloud deployment and customer success…

IBM has 15 core contributors15

IBMers working on OpenStack – from formation of the Foundation to Code Quality & New Function380

IBMers active in the projects

2

107

Mar2013

May2014859

Contributors8,500 Individuals

2130Contributors16,100+ Individuals

Exponential growth

Platinum Sponsors

OpenStack Participant Growth

Page 4: Optimizing Cloud Foundry and OpenStack for large scale deployments

4 © 2014 IBM Corporation4

©2014 IBM Corporation

…And embracing Cloud Foundry as an open source PaaS

Continuing our history of strengthening and extending open source

TOTAL CONTRIBUTORS

LINES OF CODE

PULL REQ / WK COMPANIES

1,274 Average per month2013 average: 13312mo average: 98

Lifetime average: 58

711k

50+ 43

Page 5: Optimizing Cloud Foundry and OpenStack for large scale deployments

5 © 2014 IBM Corporation

Cloud Foundry PaaS

Cloud Foundry services registry and runtime management layer.

Components are dynamically discoverable and loosely coupled, exposing health through HTTP endpoints so agents cancollect state and act on it.

Cloud Foundry architectural overview

User Authentication and Authorization

Router

DEA Pool

Service Gateway Apps

Service Connector

Health Manager

Messaging

Cloud Controller

Build Packs

Cloud Foundry BOSH

MicroClouds

PrivateClouds

PublicClouds

Service Nodes

Page 6: Optimizing Cloud Foundry and OpenStack for large scale deployments

6 © 2014 IBM Corporation

Integrate!

Page 7: Optimizing Cloud Foundry and OpenStack for large scale deployments

7 © 2014 IBM Corporation

PaaS

BOSH Cloud Provider Interface (CPI) Initially written for VMware and Amazon Web Services OpenStack CPI• OpenStack CPI written by PistonCloud Apache2 license• PistonCloud - An enterprise OpenStack company, founded by few of OpenStack

founders

Cloud Foundry – OpenStack integration

OpenStack

Cloud

Servers Networking Storage

Integration

IaaS

Hardware

Page 8: Optimizing Cloud Foundry and OpenStack for large scale deployments

8 © 2014 IBM Corporation

BOSH deployment process

Deployment Manifest• Release

name/version• # VMs, job params• Stemcells to use

Stemcell• Base OS• BOSH agent

Release• Name

Jobs• Software packages• Config templates• Scripts

BOSH

DeployedEnvironment Virtual Machine

• Configuration• Software Packages

Virtual Machine• Configuration• Software Packages

Virtual Machine• Configuration• Software Packages

Virtual Machine• Configuration• Software packages

Page 9: Optimizing Cloud Foundry and OpenStack for large scale deployments

9 © 2014 IBM Corporation

Sample manifest

Page 10: Optimizing Cloud Foundry and OpenStack for large scale deployments

10 © 2014 IBM Corporation10

Cloud Provider Interface for OpenStack

• OpenStack CPI is an implementation of the BOSH CPI.

• Leverages the Fog Ruby gem for OpenStack

• OpenStack CPI gem (Ruby package), the OpenStack Registry and Agents all reside on the Cloud Foundry Bosh repository https://github.com/cloudfoundry/bosh

• Stemcell Image (Base OS Image) and sample manifest files for OpenStack exist

Page 11: Optimizing Cloud Foundry and OpenStack for large scale deployments

11 © 2014 IBM Corporation

Requirements:• Static / floating ips• Persistent disks• Outbound Internet connectivity• Custom flavors• Increased quota• Security groups

11

Integration requirements for OpenStack

Page 12: Optimizing Cloud Foundry and OpenStack for large scale deployments

12 © 2014 IBM Corporation

Automate!

Page 13: Optimizing Cloud Foundry and OpenStack for large scale deployments

13 © 2014 IBM Corporation

OpenStack discovery:

• Extend the Fog OpenStack gem used by CPI to discover and OpenStack artifacts in an automated manner in the Cloud Foundry manifest

• Pass OpenStack credentials and discover OpenStack compute and network information.

Deployment automation: Tip one

Discover Flavors

Discover Networks

Discover VM Subnet

Discover DHCP IP

Discover Keypairs

Discover Gateway IP

Discover Security groups

Page 14: Optimizing Cloud Foundry and OpenStack for large scale deployments

14 © 2014 IBM Corporation

Deployment automation: Tip twoOpenStack pre-req setup:

• Extend the Fog OpenStack gem used by CPI to setup Cloud Foundry OpenStack pre-reqs• Setup tenant quota• Keypairs• Custom Flavors• Security Groups setup

Create Keypair

Create FlavorsRouter, DEA, Cloud Controller, Service Nodes

Create Security GroupsCf-private, cf-public

Setup tenant quota

Page 15: Optimizing Cloud Foundry and OpenStack for large scale deployments

15 © 2014 IBM Corporation

Deployment automation: Tip threeCloud Foundry manifest generation:

• Hard to manually create and edit Cloud Foundry manifests – which can extend up to 1000+ lines • Automation around template creation – an extension from the previous efforts of OpenStack Discovery

and Cloud Foundry pre-req setup. • Can create templates for both nova-network and neutron based environments• Iterates through jobs and assigns static IPs to them if we are using an environment without PowerDNS.

Page 16: Optimizing Cloud Foundry and OpenStack for large scale deployments

16 © 2014 IBM Corporation

Deployment automation: Tip fourStemcell automation:

• Community stemcell image has empty fstab and this makes the root partition auto detection to fail, and consequently provisioning to fail.

Multiple Solutions: 1. Inject file into stemcell

• Mount stemcell image using qemu-nbd• Inject /etc/fstab into mounted os• Create new image based on snapshot of modified os

2. Modify Openstack Configuration: • Change “libvirt_inject_partition” parameter in nova.conf from -1 (autodetect) to 1 (use first partition)

3. Modify bosh “Stemcell Builder” script• Script injects /etc/fstab into CentOS, but not Ubuntu (as seen here)

4. Install cloud-init in the image• Best solution to make it work in conjunction with metadata service

Stemcell• Base OS• BOSH Agent

Page 17: Optimizing Cloud Foundry and OpenStack for large scale deployments

17 © 2014 IBM Corporation

Scale!

Page 18: Optimizing Cloud Foundry and OpenStack for large scale deployments

18 © 2014 IBM Corporation

Sample CF sizing for 1,000 small applications

~60 virtual machines

• 20 high memory DEAs

• 11 CF fabric components

• 26 service broker and service nodes

160 CPU

~500 GB memory

~1.5 TB VM disk

~200 GB block storage

Page 19: Optimizing Cloud Foundry and OpenStack for large scale deployments

19 © 2014 IBM Corporation

Controller Node

Our initial OpenStack cluster was built on powerfulhardware…

…but it was not ideal for Cloud Foundry

Compute NodesCompute NodesCompute Node

Nova Compute

Nova Network

Storage Node

Cinder (volume)

Glance (image)

1x

20x

Dedicated Servers

(bare metal)

Nova API

Nova Scheduler

Keystone

Horizon

Database (MySQL)

Messaging (Qpid)

1x

128 GB memory6 cores36 TB diskeach

Page 20: Optimizing Cloud Foundry and OpenStack for large scale deployments

20 © 2014 IBM Corporation

Weaknesses in the OpenStack layout affectedthe stability of our Cloud Foundry configuration

Single points of failure in the networking, messaging, and storage configuration made BOSH unhappy.

‒ Qpid on Folsom memory leaks made the head node a bottleneck. ‒ Lost messages affected VM and storage creation, leaving many

resources in stuck states.‒ Single Nova API process design was overloaded with BOSH traffic‒ VLAN per tenant design routed much traffic through the single

head node, impacting both the fabric and brokered services.‒ Cinder and Glance storage and APIs were located on a single

node.

Bare metal allocation of resources was also inefficient‒ Every OpenStack component ran as a process directly on the OS‒ Made reacting to workload changes difficult, i.e., difficult to

redistribute OpenStack components to react to Cloud Foundry demands

Page 21: Optimizing Cloud Foundry and OpenStack for large scale deployments

21 © 2014 IBM Corporation

We’ve addressed the major issues OpenStack by distributing components more intelligently

Moving some components to VMs, assigning controller, compute, data, storage roles to groups of components improves availability, flexibility, scalability and maintainability

Compute NodesCompute NodesCompute Node

Nova Compute

Cloud Controller Data Nodedatabase (mysql)

messaging (qpid)

Cloud Controller Master Node

nova-api

nova-scheduler

Keystone

Horizon

Cloud Controller Data Nodedatabase (MySql)

messaging (Qpid)

Cloud ControllerStorage Node

Cinder (volume)

Glance (image) Nova Network

Controller Node

Nova API

Nova Scheduler

Keystone

Horizon

Storage Node

Cinder (volume)

Glance (image)

2xData Node

Database (MySQL)

Messaging (RabbitMQ)

2x 15x

Load Balancer NodeHA Proxy

Keepalive

Load Balancer NodeHA Proxy

Keepalive

Load Balancer NodeHA Proxy

KeepAlive

6x3xVirtual

Machines(VMs)

Dedicated Servers

(bare metal)

Page 22: Optimizing Cloud Foundry and OpenStack for large scale deployments

22 © 2014 IBM Corporation

Matching the right hardware for OpenStack to host Cloud FoundryLearn more about these topics at sessions from our team Wednesday

Page 23: Optimizing Cloud Foundry and OpenStack for large scale deployments

23 © 2014 IBM Corporation

Optimize!

Page 24: Optimizing Cloud Foundry and OpenStack for large scale deployments

24 © 2014 IBM Corporation

Deploy optimization tips• Increase OpenStack API rate limits (/etc/nova/api-paste.ini)

[filter:ratelimit]paste.filter_factory = nova.api.openstack.compute.limits:RateLimitingMiddleware.factorylimits = (POST, "*", .*, 9999, MINUTE);(POST, "*/servers", ^/servers, 9999, DAY);(PUT, "*", .*, 9999, MINUTE);(GET, "*changes-since*", .*changes-since.*, 9999, MINUTE);(DELETE, "*", .*, 9999, MINUTE)

BOSH

BOSHAdministrator

NATS

PostGres

Redis

Blob Store

Health Monitor

Director

Registry

Clo

ud P

rovi

der I

nter

face

(CP

I)

Cloud Foundry Environment

mongo_gw

mysql_gw

redis_gw

postgres_gw

cloud_ctrler

ccdb_ng

router

nats

debian_nfs

dea

uaa_db

uaa

loggregator

health_mgr rabbit_gw

mongo_node

mysql_node

redis_node

postgres_node

rabbit_node

OpenStackEnvironment

Ope

nSta

ck

AP

I

Page 25: Optimizing Cloud Foundry and OpenStack for large scale deployments

25 © 2014 IBM Corporation

Deploy optimization tips• Increase BOSH’s NATS timeout

• A large number of BOSH agents can overwhelm NATS• Place the following in your MicroBOSH manifest file:

apply_spec: properties: nats: ping_interval: 30 ping_max_outstanding: 30

BOSH NATS

mongo_gwmysql_gw redis_gwpostgres_gw

cloud_ctrler

ccdb_ng

router

nats

debian_nfs

dea

uaa_db

uaa

loggregator

health_mgr

rabbit_gw

mongo_nodemysql_node redis_nodepostgres_node rabbit_node

Page 26: Optimizing Cloud Foundry and OpenStack for large scale deployments

26 © 2014 IBM Corporation

Deploy optimization tips• Avoid name based security groups with nova-network

• Name based security groups require message bus activity and database updates proportional to the number of existing VMs

• Use an OpenStack scheduler which distributes VM load• Ensure that jobs are distributed across compute nodes instead of grouping• The default scheduler has the correct behavior

compute_scheduler_driver = nova.scheduler.filter_scheduler.FilterScheduler

Page 27: Optimizing Cloud Foundry and OpenStack for large scale deployments

27 © 2014 IBM Corporation

Security optimization tipsGuiding principle: Use the most limited permissions required to complete the job

1. Use tenant credentials• Do not use full admin credentials in your BOSH manifest

2. Only open the security ports which are required

Page 28: Optimizing Cloud Foundry and OpenStack for large scale deployments

28 © 2014 IBM Corporation

Security optimization tips3. Separate your VM network and OpenStack management network

• One challenge with BOSH VMs deployed on OpenStack• Setup pin holes for those particular VMs (Director, OpenStack Registry)

4. Separate Cloud Foundry’s core fabric from other brokered services• Services connected using the service broker should be on a separate network• Prevents compromised services from effecting Cloud Foundry

Page 29: Optimizing Cloud Foundry and OpenStack for large scale deployments

29 © 2014 IBM Corporation

What next? Join us at the Cloud Foundry Summit

Page 30: Optimizing Cloud Foundry and OpenStack for large scale deployments

30 © 2014 IBM Corporation

Monday, May 12 – Room B314

12:05-12:45

Wednesday, May 14 - Room B312

9:00-9:40

9:50-10:30

11:00-11:40

11:50-12:30

OpenStack is Rockin’ the OpenCloud Movement! Who‘s Next to Join the Band ?Angel Diaz, VP Open Technology and Cloud LabsDavid Lindquist, IBM Fellow, VP, CTO Cloud & Smarter Infrastructure

Getting from enterprise ready to enterprise bliss - why OpenStack and IBM is a match made in Cloud heaven. Todd Moore - Director, Open Technologies and Partnerships

Taking OpenStack beyond Infrastructure with IBM SmartCloud Orchestrator.Andrew Trossman - Distinguished Engineer, IBM Common Cloud Stack and SmartCloud Orchestrator

IBM, SoftLayer and OpenStack - present and futureMichael Fork - Cloud Architect

IBM and OpenStack: Enabling Enterprise Cloud Solutions Now.Tammy Van Hove -Distinguished Engineer, Software Defined Systems

IBM Sponsored Sessions

Page 31: Optimizing Cloud Foundry and OpenStack for large scale deployments

31 © 2014 IBM Corporation

Monday, May 12

3:40 - 4:20

3:40 - 4:20

Tuesday, May 13

11:15 - 11:55

2:00 - 2:40

5:30 - 6:10

5:30 - 6:10

Wednesday, May14

9:50 - 10:30

2:40 - 3:20

Thursday, May 15

9:50 - 10:30

1:30 - 2:10

2:20 - 3:00

IBM Technical Sessions

Page 32: Optimizing Cloud Foundry and OpenStack for large scale deployments

32

Be sure to stop by the IBM booth to see some demos and get your rockin’ OpenStack t-shirt while they last.

Thank you !