cloud foundry and openstack - world`s top two open source projects!

Try IBM Bluemix for free, follow this link https://ibm.biz/Meetup_BayArea

http://www.meetup.com/OpenStack http://www.meetup.com/CloudFoundry

Cloud Foundry & OpenStack – World`s Top 2 Open Source Cloud Projects!

Apps

@AnimeshSingh


Cloud Foundry & OpenStack - Top Two Open Source Cloud Technologies!

http://analystpov.com/cloud-computing/top-15-open-source-cloud-computing-technologies-2014-24727


OpenStack’s Phenomenal Growth COMPANIES

ACTIVE MEMBERS 461 18152

COMMITS

130K

Source: http://www.ohloh.net/p/openstack

2013 OCT 17 Release: Havana 1,729,137 lines of code

2011 SEP 22 Release: Diablo 478,671 lines of code

2014 APR 17 Release: Icehouse 1,766,546 lines of code

LINES OF CODE

2M

2013 SEP 27 Release: Folsom 667,895 lines of code 2012 APR 5

Release: Essex 558.368 lines of code

2013 APR 4 Release: Grizzly 1,323,479 lines of code

2011 APR 3 Release: Cactus 117,887 lines of code

2011 FEB 3 Release: Bexar 85,425 lines of code

2010 OCT 21 Release: Austin 52,796 lines of code

2014 OCT 16 Release: Juno 2,084,291 lines of code


OpenStack Overview: Collection of well integrated IaaS modules:

•  Compute (Nova) •  Networking (Neutron) •  Object Storage (Swift) •  Block Storage (Cinder) •  Identity(Keystone) •  Image Service (Glance) •  Dashboard (Horizon)


Meets Developer’s Needs Focus on app development, not provisioning VMs, databases, messaging servers, etc Agile development model Deploy and scale in seconds

Open Cloud Platform There is an increasing appetite for cloud-based mobile, social and analytics applications from line-of-business executives - drives the need for a more open cloud development platform

Compelling Community Cloud Foundry has a compelling community and emerging ecosystem as well as a mature set of capabilities and robustness

Platinum Founding Sponsors 1.3k 800k LINES OF CODE TOTAL CONTRIBUTORS

Cloud Foundry Overview


Eclipse IDE

CLI

Browser

cf push

h4p

Cloud Foundry Developer Experience


Eclipse IDE

CLI

Browser

cf push

h4p

Cloud Controller (API)

Router

DEA Pool DEA Pool Droplet Execu>on

Agent (DEA) VM Pool

Health Manager

NATS (Message bus)

Cloud Foundry Architecture


Integrate !


PaaS

Cloud

Integration

IaaS

UAA

Router

DEA Pool Apps

Service Connector

Health Manager

Messaging

Cloud Controller

Build Packs

Service Nodes

BOSH Cloud Provider Interface

Cloud Foundry , OpenStack Integration

•  Static / floating ips •  Persistent disks •  Custom VM

Configurations •  Specialized Security

groups • 


BOSH deployment process

Deployment Manifest •  Release name/version •  # VMs, job params •  Stemcells to use

Stemcell •  Base OS •  BOSH agent

Release •  Name •  Software packages •  Config templates •  Scripts

BOSH Deployed Cloud Foundry

Virtual Machine •  Configuration •  Software Packages



Virtual Machine •  Configuration •  Software packages


Sample manifest


Open, by design Building on open from the ground up enables IBM to maximize client investment

Hybrid Cloud Private Cloud Off-premise cloud

IaaS

PaaS

SaaS OAuth


IBM Contributions to OpenStack: Road To Juno

Source http://www.stackalytics.com/

Essex

54

Core Contributors: 1 Technical Contributors: 2 Commits: 9 Blueprints: 0 Projects: 6

Folsom


100 Grizzly

270


Havana


380

Total IBMers

Key Contribu8ons: Chinese Transla>on Nova Hygiene Storage Enhancements

Total IBMers

Key Contribu8ons: Integra>on Tests Crowd Sourced Transla>on Membership Services

Key Contribu8ons: API Stability Storage Enhancements 21% of Nova design features Total IBMers

Total IBMers

Key Contribu8ons: Enterprise Security Ceilometer Quality Assurance

Icehouse

380

Core Contributors: 14 Technical Contributors: 107 Commits: 1722 Blueprints: 85 Projects: 61 Key Contribu8ons: Quality Assurance Authen>ca>on & Security 15% of Compute features Total IBMers

Juno

Core Contributors: 15 Technical Contributors: 109 Commits: 1669 Blueprints: 48 Projects: 78 Key Contribu8ons: Federated Iden>ty Block Volume Replica>on Dashboard Enhancements 400 Total IBMers


IBM BlueMix (powered by Cloud Foundry) http://bluemix.net

§  IBM initiative to develop a Platform as a Services offering

§  IBM and partner cloud services

§  Integrated DevOps with both Browser and Eclipse-based tools

Services

Lifecycle Management (JazzHub)

Applica>on Run>me

Run>mes & Frameworks

Middleware Applica>on Opera>onal Mobile External Data

node java ruby Worklight WebSphere Liberty

Web IDE (Eclipse Orion) Eclipse IDE

Applica>on Composi>on Environment

Create & Manage Services

Test/Run Test/Run

Explore Services

Explore Services

IBM Bluemix ™ Check In Code Check In Code


Bluemix allows developers to focus on the code Cloud Foundry provides unprecedented flexibility & power

Design Thinking

Containers

Extreme Agile

Mobile IoT

APIs

Microservices


OpenStack Swift Object Storage Service Available in Bluemix!


Docker Container Service running on OpenStack Available in Bluemix!


Integrate !


PaaS

Cloud

Integration

IaaS

UAA

Router

DEA Pool Apps

Service Connector

Health Manager

Messaging

Cloud Controller

Build Packs

Service Nodes

BOSH Cloud Provider Interface

Cloud Foundry , OpenStack Integration

•  Static / floating ips •  Persistent disks •  Custom VM

Configurations •  Specialized Security

groups • 


BOSH deployment process

Deployment Manifest •  Release name/version •  # VMs, job params •  Stemcells to use

Stemcell •  Base OS •  BOSH agent

Release •  Name •  Software packages •  Config templates •  Scripts

BOSH Deployed Cloud Foundry




Virtual Machine •  Configuration •  Software packages


Sample manifest


Automate the Lifecycle (Leverage the power of community) !


Leverage the power of Open Source Community

•  Open technologies give us a tremendous amount of power and flexibility to integrate them together seamlessly

•  Lets see some of the examples of what we can do with the power of Open technologies to go from bare metal hardware to a deployed Cloud Foundry ready to push apps

Open Source for Cloud Foundry and OpenStack Automation!


OpenStack Installation:

•  Leverage the open source Chef Cloud infrastructure Automation framework •  Requires information about hardware, network environment and software repositories. •  Automate OpenStack installation in approximately 15 minutes!

Chef for OpenStack Install Automation

Setup Compute

Setup Network

Setup Controller

Setup Storage


OpenStack Discovery:

•  Leverage the open source Fog gem to discover OpenStack artifacts in an automated manner •  Require OpenStack credentials and discover OpenStack compute and network information.

Fog for OpenStack Discovery Automation

Discover VM Configuration Sizes

Discover Network Subnets

Discover Network Security Rules

Discover DHCP , DNS Gateway and floating IPs

Discover Security Credentials


Fog for OpenStack Setup Automation Cloud Foundry Pre-req setup on OpenStack:

•  Leverage the open source Fog gem to setup Cloud Foundry requirements in an automated manner •  Setup according to best practices and guidelines – still giving users the flexibility to change if desired

Create Security Credentials

Create VM configs for Router, DEAs, Cloud Controller, Service Nodes

Create network Security Rules

Setup tenant quota


Community stemcell image had empty fstab and this makes the root partition auto detection to fail

1.  Inject file into stemcell

Mount stemcell image using qemu-nbd - Inject /etc/fstab into mounted os 2.  Or Modify Openstack Configuration:

Change “libvirt_inject_partition” parameter in nova.conf from -1 (autodetect) to 1 (use first partition) 3. Or Install cloud-init in the image

Best solution to make it work in conjunction with metadata service

Modify Stemcells

Techniques for Stemcell Automation


BOSH and Ruby for Cloud Foundry Deployment Automation Cloud Foundry Deployment Automation

•  Automate base OS image creation or modification •  Automate Cloud Foundry deployment manifest file genration using Ruby ERB •  Automate upload of Cloud Foundry core release, services and runtime frameworks, followed by Cloud

Foundry deployment

Stemcell Creation and Upload

Generate BOSH and Cloud Foundry Manifest

Upload Cloud Foundry core, Services and runtime

Deploy Cloud Foundry

Deploy Microbosh

RUBY BOSH


Automate end to end deployments in under an hour !

•  Previous example signify the power of Open source community •  Leveraging tools like Chef, Fog, BOSH, Ruby ERB from the community can help create powerful

automation which can be repeatedly and consistently replicated in under an hour!

Open way is the only way!

BOSH

RUBY


Automate the Update/Upgrade!


Automate the update/upgrade lifecycle operations •  Updates can be automated using code from the initial automated deployment (e.g. bosh deploy) •  Follow te same workflow – do OpenStack discovery operation, and then leverage BOSH for update/

upgrade/ •  To ensure application availability throughout the update, use tools like JMeter to test application responsiveness

Updates/Upgrades Automation!


Application Uptime during Updates/Upgrades Automation!


Scale and Optimize


Sample CF sizing for 1,000 small applications

~60 virtual machines •  20 high memory

DEAs

•  11 CF fabric components

•  26 service broker and service nodes

160 CPU ~500 GB memory ~1.5 TB VM disk ~200 GB block storage


Scaling OpenStack Optimize Internal Communication

•  Configure OpenStack for scaled concurrency

Optimize Performance •  Configure OpenStack scheduler to evenly distribute load

Setup Highly Available Architecture for PaaS workloads

Load Balancer Nodes

Data Nodes

3x

Compute Nodes

15x Storage Nodes

2x

Controller Nodes

2x 6x


Scaling Cloud Foundry / BOSH Optimize Internal Communication

•  Configure messaging bus for VM communication

Optimized routing and bandwidth allocation •  Isolate Cloud Foundry components using multiple networks

Maintain Cloud Foundry’s Highly Available Architecture

Service Gateways

10x Routers 3x

Service Nodes 15x

DEAs 20x

Cloud Controllers

2x


Deploy optimization tips

•  Increase OpenStack API rate limits (/etc/nova/api-paste.ini)

[filter:ratelimit] paste.filter_factory = nova.api.openstack.compute.limits:RateLimitingMiddleware.factory limits = (POST, "*", .*, 9999, MINUTE); (POST, "*/servers", ^/servers, 9999, DAY); (PUT, "*", .*, 9999, MINUTE); (GET, "*changes-since*", .*changes-since.*, 9999, MINUTE); (DELETE, "*", .*, 9999, MINUTE)

BOSH

BOSH Administrator

NATS

PostGres

Redis

Blob Store

Health Monitor

Director

Registry

Clo

ud P

rovi

der I

nter

face

(C

PI)

Cloud Foundry Environment

mongo_gw

mysql_gw

redis_gw

postgres_gw

cloud_ctrler

ccdb_ng

router

nats

debian_nfs

dea

uaa_db

uaa

loggregator

health_mgr rabbit_gw

mongo_node

mysql_node

redis_node

postgres_node

rabbit_node

OpenStack Environment O

penS

tac

k A

PI


Deploy optimization tips •  Increase BOSH’s NATS timeout

•  A large number of BOSH agents can overwhelm NATS •  Place the following in your MicroBOSH manifest file: apply_spec: properties: nats: ping_interval: 30 ping_max_outstanding: 30

BOSH NATS

mongo_gw mysql_gw redis_gw postgres_gw

cloud_ctrler

ccdb_ng

router

nats

debian_nfs

dea

uaa_db

uaa

loggregator

health_mgr

rabbit_gw

mongo_node mysql_node redis_node postgres_node rabbit_node


Security optimization tips

Separate your VM network and OpenStack management network •  One challenge with BOSH VMs deployed on OpenStack •  Setup pin holes for those particular VMs (Director, OpenStack Registry)

Separate Cloud Foundry’s core fabric from other brokered services

•  Services connected using the service broker should be on a separate network •  Prevents compromised services from effecting Cloud Foundry


Why Cloud Foundry and OpenStack are a great fit ?

•  100% Open PaaS and IaaS solutions – No vendor lock-ins •  Strong and growing community of contributors and sponsors on both sides

•  Power of Open Source community can be leveraged to automate the deployment and

lifecycle management of Cloud Foundry on OpenStack

•  OpenStack meets Cloud Foundry integration requirements, and is totally configurable and adaptable to handle the scale of a PaaS solution like Cloud Foundry

•  Bottom Line: They are a great fit!


What next? Join us at our Silicon Valley Meetups

http://www.meetup.com/BlueMix/

http://www.meetup.com/CloudFoundry/

For more technical details, refer to the slides and video here http://www.slideshare.net/AnimeshSingh/optimizing-cloud-foundry-and-openstack-for-large-scale-deployments https://www.openstack.org/assets/presentation-media/A-Practical-Approach-to-HA-Final.pdf https://www.youtube.com/watch?v=jCwtV9n_ak4 https://www.youtube.com/watch?v=FsrCGkBo4Vg

cloud foundry and openstack - world`s top two open source projects!

Technology

ibm bluemix

lines of code lines

link https

cloud foundry openstack

open source cloud projects

open source cloud technologies

bayarea http

seconds open cloud platform