© 2013 SpringOne 2GX. All rights reserved. Do not distribute without permission.

Free Yourself with Cloud FoundryA Private Cloud Experience

Mike Heath, Shawn Nielsen, Mike Youngstrom

Disclaimer

This presentation does not represent the views, opinions,

policies, nor direction of The Church of Jesus Christ of

Latter-day Saints. These views are the sole views of the

presenters involved with the presentation. We take full

responsibility for content presented and any errors or

incorrect perception of representation.

Description / Overview

• This presentation is about

our experience integrating

Cloud Foundry into our

organization.

• It is not a tutorial on how

to deploy CF.

Presentation Roadmap

• Why we chose Cloud Foundry?

• Cloud Foundry APIs in a UI

• Gap analysis

• Develop and deploy

• Authentication & diagnostics

• Custom services

• Application level firewalls

Business Problems

• 100s of small/medium apps

• Difficult to manage infrastructure for so

many apps

• More interested in fault tolerance over

scale for most apps

• Slowness in provisioning time

The PaaS Team

• 2 ½ Developers

• ½ Manager

• 3 Operations personnel

• Passion for efficiency

Current State

• Currently in beta

– Moving to enterprise ready in Q4

• Good feedback so far

• Applications

– 4 production

– 14 development

Why we went with PaaS

• Developer productivity

• Fault tolerant and scalable

• Simplification of infrastructure

• Consistent deployments across runtimes

• Self-service

Why Cloud Foundry?

Other Reasons for Cloud Foundry

• Infrastructure agnostic

– Avoids vendor lock-in

• Public/private

• Cloud on cloud

– PaaS running on an IaaS cloud

Other Reasons We Love Cloud Foundry

• Open source

– Ability to adapt Cloud Foundry to our needs

– Community involvement

• Architecture

– Good architecture sells itself

Overview of Environment

• vSphere

• F5 (load balancer)

• Oracle

• LDAP

• Proxy based SSO

• Primarily Java+Spring shop

– Starting to see some NodeJS

Cloud Foundry Adoption Challenges

Cloud Foundry Adoption Challenges

• Convincing the systems engineers

– VM centric world to app centric world

Network Zoning and Firewalls

• Network firewalls

• Transition to host / app firewalls

Network Firewalls

Host / App Level Firewalls

Cost Challenges

• Current costs and billing models

– Multi-year bill-out

• How do you know what you’re going to use in 4 years?

You don’t!

Quota and Utilization Bill-back

Know your bill

$25 per GB RAM per month

Know your usage

Trade-in Model

Organizational Efficiencies

Presentation Roadmap

• Why we chose Cloud Foundry?

• Cloud Foundry APIs in a UI

• Gap analysis

• Develop and deploy

• Authentication & diagnostics

• Custom services

• Application level firewalls

Billing and Usage Through CC APIs

CloudController API:

url: "/v2/quota_definitions/36673f76-c617-4ae8-94b9-7adccb747ced”

entity: {

name: "Enterprise Management Organization - Quota“

non_basic_services_allowed: false

total_services: 100

memory_limit: 40960

trial_db_allowed: false

}

Custom UI

• Just like everyone else…we have our own UI.

• Why?

– No standard UI available. Though several community UIs on

their way.

– No way to do access management through CF. Why Not?

– Management of quotas for organizations

– Give management something visual to look at

Cloud Foundry APIs

CC URL: /v2/organization/<guid>

entity: {

name: “MySpringOrganization"

quota_definition_url: "/v2/quota_definitions/<guid>

spaces_url: "/v2/organizations/<guid>/spaces”

domains_url: "/v2/organizations/<guid>/domains”

users_url: "/v2/organizations/<guid>/users”

managers_url: "/v2/organizations/<guid>/managers”

auditors_url: "/v2/organizations/<guid>/auditors”

app_events_url: "/v2/organizations/<guid>/app_events”

}

s

Key Performance Indicator (KPI) Data

Quota

state: "RUNNING"

stats: {

uptime: 984643

mem_quota: 536870912

disk_quota: 1073741824

fds_quota: 16384

}

Live Usage

usage: {

time: "2013-09-10 02:05:27”

cpu: 0.0007454006633748

mem: 211116032

disk: 116498432

}

Cloud Foundry APIsCC URL: /v2/apps/<guid>/stats

Operations Center

UI Demo

Presentation Roadmap

• Why we chose Cloud Foundry?

• Cloud Foundry APIs in a UI

• Gap analysis

• Develop and deploy

• Authentication & diagnostics

• Custom services

• Application level firewalls

Gaps

Authentication

Diagnostics

Enterprise Services and Legacy

Presentation Roadmap

• Why we chose Cloud Foundry?

• Cloud Foundry APIs in a UI

• Gap analysis

• Develop and deploy

• Authentication & diagnostics

• Custom services

• Application level firewalls

Deployment

• Use BOSH– Fork cf-release

– Proxy?

• Environments:– X Dev

– 1 Test

– 1 Prod

• Customers only use prod

• Break deployments into pools– Core

– DEA/Router-X

Development

• Dev is prod but smaller– Use BOSH

– vSphere

– F5

– SSL – kind of

• Develop on component?– `bosh stop` existing component

– Configure local component in place

• Other options– Vagrant?

– Warden CPI?

Presentation Roadmap

• Why we chose Cloud Foundry?

• Cloud Foundry APIs in a UI

• Gap analysis

• Develop and deploy

• Authentication & diagnostics

• Custom services

• Application level firewalls

Customizing Authentication

• UAA & Login

– Java Spring + Spring Security

• Customized UAA

– Added UAA to src release

– Added spring config

• Login Server

– May re-visit

Buildpack Customization

• Vital extension point

• Support for legacy artifacts

• Just fork and tweak

– Proxies

– Oracle support for Node

– Pre-processed `npm install`

– Add JMX support

Problem: Diagnostics

• Need:– Thread dumps

– Heap dumps

– Remote debug

– APM

• Current story? Not good– Logging

– Console port???

– Debug port???

Solution: Caldecott?

• Use console and debug ports

• Caldecott

– TCP over HTTP proxy

– Inter-app communication

• Diagnostic

– Router/gateway/server

Solution: Diagnostics Server

Dev Workstation

Diagnostic Server

UAA/Cloud Controller

DEA

Warden

Request Proxy

Authorize Request

Proxy to DEA

Forward to Warden onDebug or Console Port

Demo Remote Debug

JMX

• JMX is great– Heap dumps

– Thread dumps

– Basic profiling

– Management operations

• JMX over RMI lacking– JMXMP to the rescue

• Configure JMX in buildpack– Bind to `console` port

• Created `cf` plugin to launch Visualvm

JMX Demo

Diagnostics Summary

• Caldecott

– Doesn’t work for us

• Diagnostic Server

• Safe customization?

– Removed ports

Presentation Roadmap

• Why we chose Cloud Foundry?

• Cloud Foundry APIs in a UI

• Gap analysis

• Develop and deploy

• Authentication & diagnostics

• Custom services

• Application level firewalls

Custom Services

• Integrate with existing systems

– LDAP, NFS/CIFS, Oracle, SSO,

Web Services, etc.

• New ‘user-provided’ service is not

adequate

– Will still work for most people

– We need more than a set of

key/value pairs

Custom Services Built on Java and Spring

• We’re a Java shop.

• Java is more enterprise

friendly.

– Client libraries for existing

systems

• Our team consists of

Java/Spring developers.

Custom Service Framework Features

• NATS Client

• Cloud Foundry utilities

– Type-safe NATS messaging

– PID file support for working with Monit/BOSH

– YAML config

• Cloud Controller client for invoking service APIs

• Service Broker

• Open source

– https://github.com/cloudfoundry-community/java-nats

– https://github.com/cloudfoundry-community/cf-java-component

NATS Java Client

• NATS is the distributed pub/sub messaging system used

by Cloud Foundry.

• NATS support is essential for customizing Cloud Foundry.

• Java client

– Built using Netty

– Integrates with Spring

• Generic NATS client, nothing Cloud Foundry specific

NATS Client Sample Code

// Connecting NATS to server

Nats nats = new NatsConnector()

.addHost("nats://localhost")

.connect();

// Simple subscriber

nats.subscribe("foo", (message) -> {

System.out.println("Received: " + message);

});

// Simple publisher

nats.publish("foo", "Hello world!");

NATS Client Spring Integration

<nats:nats>

<nats:url>nats://localhost:4222</nats:url>

</nats:nats>

@EnableNatsAnnotations

@Configuration

public class Configuration {

@Subscribe("foo")

public void onMessage(Message message) {

System.out.println(message);

}

}

Spring Boot/NATS Example

• NATS to HTTP Gateway

– Using Spring Boot

– Single Java source file

– Less-than 75 lines of Java (including import statements.)

https://github.com/mheath/spring-boot-nats-example

Cloud Foundry Specific NATS (cf-nats)

• Simplifies using NATS with Cloud Foundry

• Type-safe NATS messaging

– Component discovery

– Router registration

– Staging notifications

• Still a work in progress

https://github.com/cloudfoundry-community/cf-java-component/tree/master/cf-nats

PID File

• Support for creating .pid file at Spring context initialization

• Useful for working with Monit

– BOSH uses Monit

<cf:pid-file resource=“file:/var/run/component/my-cf-component.pid” />

YAML Configuration

• Existing Cloud Foundry components use YAML for

configuration.

• YAML is cool.

• We provide Spring integration for using YAML.

– XML configuration for loading YAML as properties.

– Java configuration for using YAML is a Spring

PropertySource.

Service Broker Framework

• Enables creating custom services

• Creating service broker requires implementing a single

Java interface

• Simple Cloud Controller client for invoking service APIs

• Provides APIs for automatically registering with Cloud

Controller

Example Service Broker

• Built on Spring Boot

• http://github.com/cloudfoundry-community/java-service-

broker-example

Example Service Broker Demo

Oracle Service Demo

Custom Oracle Service

• Prompts for: service name, schema/user, password

• Produces credentials:"credentials“ : {

"schema":"CF_DEV2",

"ldap“ : {"host":“fake-ldap.lds.org", "port":389, "context":"cn=OracleContext…"},

"firewall":[{"port":1234,"host":“oracle-scan-host”}…],

"descriptor":"(DESCRIPTION=(ADDRESS_LIST=…",

"alias":“DB-DEV2",

"service":“service134",

"addresses":[{"host":“oracle-scan-host","port":"1234"}],

"jdbcUrl":"jdbc:oracle:thin:@ldap://fake-ldap/DB-DEV2,cn=OracleContext,…",

"password":"super-secret-password"}}

Custom Service Creation Workflow

`cf` client

Service Broker

Cloud Controller

cf plugin prompts

for service data

cf plugin sends data

to broker (host:

‘foo’, port: 1234)

cf creates service

Cloud Controller tells broker

to create service instance.

Broker correlates service data

originally sent by cf plugin and

returns service credentials to

Cloud Controller.

Holds service data

(host: ‘foo’, port 1234)

Presentation Roadmap

• Why we chose Cloud Foundry?

• Cloud Foundry APIs in a UI

• Gap analysis

• Develop and deploy

• Authentication & diagnostics

• Custom services

• Application level firewalls

Firewall Requirements

• Proxy based SSO requires control over incoming requests

• Minimize attack vectors from compromised/malicious apps

– Limit access to console and debug ports

• Allow access to high risk network zones

• Auditability – What network resources does each app

have access to?

• Minimize customization of Cloud Foundry

Proxy Based Single Sign-On (SSO)

Warden

• Used by DEA to securely host applications

• Applications run with a “Warden Container”

• Warden Containers isolate resources– Network

– Disk

– Memory

• Think of a Warden Container as a “lightweight VM.”

Controlling Incoming Requests

• Load Balancer and Cloud Foundry routers easily secured.

• Securing requests to Warden Containers more difficult

– Need to allow incoming requests from routers

– Disallow outgoing requests directly to routers

• Cloud Foundry scales dynamically

– Routers can be added/removed from the system at any time

– Changing static configuration of all DEAs when router is

added or removed is unacceptable

Custom DEA Incoming Firewall

• Configure Warden to drop all incoming and outgoing

packets

• Customized DEA to track Cloud Foundry routers

– Routers already broadcast presence via NATS

– Use existing NATS messages to build router registry

• Warden uses `iptables` to isolate network resources

– `iptables` are Linux kernel level network rules

– Our firewalls piggybacks on Warden’s `iptables` rules

Custom DEA Outbound Firewall

• Service credentials

embed firewall rules

• Use Warden API to

open firewall holes

• Facilitated by custom

services

"credentials" : {

...

"firewall" : [{

"network" : "10.118.50.0/24",

"port" : 8080

}]

}

Securing Outbound HTTP

• Problems– Single IP address can host

multiple web sites/applications

– Public web services often have large pool of IP addresses that can change over time

• Solution – HTTP proxy for hostname validation

• Custom service firewall rules enables access to the HTTP proxy

"credentials" : {..."firewall" : [{“http" : “http://www.google.com"}]}

Http Proxy Firewall Workflow

HTTP Proxy

DEA

Warden Proxies HTTP requests

Network Firewall

Intranet

Internet

Summary

• Cloud Foundry is awesome!

• Cloud Controller APIs make creating a UI easy.

• Cloud Foundry authorization customization is maturing.

• Diagnostics need work. Our solution was promising.

• You can create services in Java.

• We isolate our applications in app firewalls

Questions?

Citation references• Cloud picture (slide 3) Copyright 2007, Karin Dalziel. Creative Commons licensed.

• Snail picture (slide 4) Copyright 2013, Eirien (Ilweranta). Create Commons licensed.

• Wrench photo (slide 5) Copyright 2009, zzpza. Creative Commons licensed.

• Building photo (slide 10) Copyright 2010, Ricardo Diaz. Creative Commons licensed.

• Lightening road screen do not enter picture (slide 12) Copyright 2008, Bobby. Creative Commons licensed.

• Gap (slide 28) Copyright 2013, Upupa4me. Creative Commons licensed.

• “Key Note” (slide 29) Copyright 2009, William Neuheisel. Creative Commons licensed.

• “Angry Computer Support Worker Baning His Fists On His Desk” (slide 30) Copyright 2010, jfcherry. Creative Commons licensed.

• “24 Hour Service” (slide 31) Copyright 2008, Natalie Maynor. Creative Commons licensed.

• “Launch of Atlas V TDRS-K from Cape Canaveral AFS” (slide 32) Copyright 2013 NASA Goddard Space Flight Center. Creative Commons licsened.

• “American Architect, 1909” (Slide 33) Copyright 2011 REVIVALthedigest. Creative Commons licensed.

• “Keys” (Slide 34) Copyright 2005 mmarchin. Create Commons licensed.

• “Lego building” (Slide 35) Copyright 2006 Matt Bateman. Creative Commons licensed.

• “Heavy Metal: TDK MA-R90 Cassette Tape” (Slide 39) Copyright 2012 Scott Schiller. Creative Commons licensed.

• “Tunnel” (Slide 41) Copyrigh 2010 Dushan Hanuska. Creative Commons licensed.

• “Roadside Service” (Slide 42) Copyright 2012 Pam Morris. Creative Commons licensed.

• “Typical Prison Guard” (Slide 60) Copyright 2009 Son of Groucho. Creative Commons licensed.