decomposing applications for deployability and scalability #springone2gx #s12gx

Decomposing applications for deployability and scalability

Chris Richardson, Author of POJOs in Action, Founder of the original CloudFoundry.com @crichardsoncrichardson@vmware.comhttp://plainoldobjects.com/

Presentation goal

2

How decomposing applications improves deployability and

scalability and

How Cloud Foundry helps

About Chris

3

(About Chris)

4

About Chris()

5

About Chris

6

About Chris

http://www.theregister.co.uk/2009/08/19/springsource_cloud_foundry/

7

vmc push About-Chris

8

Developer Advocate

Signup at http://cloudfoundry.com

9

AgendaThe (sometimes evil) monolithDecomposing applications into servicesHow do services communicate?Presentation layer designHow Cloud Foundry helps

Let’s imagine you are building an e-commerce application

10

Tomcat

Traditional web application architecture

11

Browser

WAR

MySQL Database

ShippingService

AccountingService

InventoryService

StoreFrontUI

developtestdeploy

Simple to

Apache

scale

But there are problems with a monolithic architecture

12

Users expect a rich, dynamic and interactive experience

13

Java Web ApplicationBrowser

HTTP Request

HTML/Javascript

Old style UI architecture isn’t good enough

Real-time web ≅ NodeJS

Intimidates developers

14

Obstacle to frequent deploymentsNeed to redeploy everything to change one componentInterrupts long running background (e.g. Quartz) jobsIncreases risk of failure

Fear of change

Updates will happen less oftene.g. Makes A/B testing UI really difficult

15

Overloads your IDE and container

16Slows down development

17

Shipping team

Accounting team

Engineering

Obstacle to scaling development

E-commerce application

18

WAR

Shipping

Accounting

InventoryService

StoreFrontUI

Shipping team

Accounting team

Inventory team

UI team

Obstacle to scaling development

19

Lots of coordination and communication required

Obstacle to scaling developmentUI team Accounting team

20

Requires long-term commitment to a technology stack

21

AgendaThe (sometimes evil) monolithDecomposing applications into servicesHow do services communicate?Presentation layer designHow Cloud Foundry helps

22

The scale cube

23

X axis - horizontal duplication

Z axi

s - da

ta pa

rtitio

ning

Y axis - functionaldecomposition

Scale

by sp

litting

simila

r

thing

s

Scale by splitting different things

Y-axis scaling - application level

24

WAR

ShippingService

AccountingService

InventoryService

StoreFrontUI

Y-axis scaling - application level

25

Store front web application

shipping web application

inventory web application

Apply X axis cloning and/or Z axis partitioning to each service

ShippingService

AccountingService

InventoryServiceStoreFrontUI

accounting web application

Partitioning strategiesPartition by verb, e.g. shipping servicePartition by noun, e.g. inventory serviceSingle Responsibility PrincipleUnix utilities - do one focussed thing well

26

Something of an art

Real world examples

27

http://highscalability.com/amazon-architecture

Between 100-150 services are accessed to build a page.

http://techblog.netflix.com/

http://www.addsimplicity.com/downloads/eBaySDForum2006-11-29.pdf

http://queue.acm.org/detail.cfm?id=1394128

There are drawbacks

28

Complexity

29

See Steve Yegge’s Google Platforms Rant re Amazon.com

Multiple databases =

Transaction management challenges

30

When to use it?

31

In the beginning: •You don’t need it •It will slow you down

Later on:•You need it•Refactoring is painful

But there are many benefitsScales development: develop, deploy and scale each service independentlyUpdate UI independentlyImproves fault isolationEliminates long-term commitment to a single technology stack

32

Modular, polyglot, multi-framework applications

Two levels of architecture

33

System-level

ServicesInter-service glue: interfaces and communication mechanismsSlow changing

Service-level

Internal architecture of each serviceEach service could use a different technology stackPick the best tool for the jobRapidly evolving

If services are small...Regularly rewrite using a better technology stackAdapt system to changing requirements and better technology without a total rewritePick the best developers rather than best <pick a language> developers ⇒ polyglot culture

34

Fred George “Developer Anarchy”

The human body as a system

35

50 to 70 billion of your cells die each day

36

Yet you (the system) remain you

37

Can we build software systems with these characteristics?

38

http://dreamsongs.com/Files/WhitherSoftware.pdf

http://dreamsongs.com/Files/DesignBeyondHumanAbilitiesSimp.pdf

39

AgendaThe (sometimes evil) monolithDecomposing applications into servicesHow do services communicate?Presentation layer designHow Cloud Foundry helps

Inter-service communication optionsSynchronous HTTP ⇔ asynchronous AMQP

Formats: JSON, XML, Protocol Buffers, Thrift, ...

Even via the database

40

Asynchronous is preferred

JSON is fashionable but binary format is more efficient

StoreFrontUI

wgrus-store.war

AccountingService

wgrus-billing.war

InventoryService

wgrus-inventory.war

ShippingService

wgrus-shipping.war

MySQL

41

RabbitMQ(Message Broker)

Asynchronous message-based communication

BenefitsDecouples caller from serverCaller unaware of server’s coordinates (URL)Message broker buffers message when server is down/slow

42

DrawbacksAdditional complexity of message broker RPC using messaging is more complex

43

Writing code that calls services

44

The need for parallelism

45

Service A

Service B

Service C

Service D

b = serviceB()

c = serviceC()

d = serviceD(b, c)

Call in parallel

Java Futures are a great concurrency abstraction

46

http://en.wikipedia.org/wiki/Futures_and_promises

Using Java Futures

47

public class Client {

private ExecutorService executorService; private RemoteServiceProxy remoteServiceProxy;

public void doSomething() throws ... { Future<Integer> result =

executorService.submit(new Callable<Integer>() { @Override public Integer call() throws Exception { return remoteServiceProxy.invokeRemoteService(); } });

/// Do other things

int r = result.get(500, TimeUnit.MILLISECONDS);

System.out.println(r);

}}

Eventually contains result

When needed wait for result

Akka’s composable futures are even better

48

Composable Futures

49

val f1 = Future { ... ; 1 }val f2 = Future { ... ; 2 }

val f4 = f2.map(_ * 2)assertEquals(4, Await.result(f4, 1 second))

val fzip = f1 zip f2assertEquals((1, 2), Await.result(fzip, 1 second))

http://doc.akka.io/docs/akka/2.0.1/scala/futures.html

Transforms Future

Combines two futures

Using Akka futures

50

def callB() : Future[...] = ...def callC() : Future[...] = ...def callD() : Future[...] = ...

val future = for { (b, c) <- callB() zip callC(); d <- callD(b, c) } yield d

val result = Await.result(future, 1 second)

http://doc.akka.io/docs/akka/2.0.1/scala/futures.html

Two calls execute in parallel

And then invokes D

Get the result of D

Spring IntegrationProvides the building blocks for a pipes and filters architectureEnables development of application components that are•loosely coupled•insulated from messaging infrastructureMessaging defined declaratively

51

Handling failure

52

Service A Service B

Errors happen in distributed systems

About Netflix

53

> 1B API calls/day1 API call ⇒ average 6 service calls

Fault tolerance is essential

http://techblog.netflix.com/2012/02/fault-tolerance-in-high-volume.html

How to run out of threads

54

Tomcat

Execute thread pool

HTTP Request

Thread 1

Thread 2

...

Thread N

Service A Service B

If service B is down then thread will be

blocked

XXXXX

Eventually all threads will be blocked

Use timeouts and retries

55

Never wait forever

Errors can be transient ⇒ retry

http://techblog.netflix.com/2012/02/fault-tolerance-in-high-volume.html

Service A

Service B

Use per-dependency bounded thread pool

56http://techblog.netflix.com/2012/02/fault-tolerance-in-high-volume.html

Runnable 1

Runnable 2

Runnable ...

bounded queue

Task 1

Task 2

Task ...

bounded thread pool

Limits number of outstanding requests

Fails fast if service is slow or down

Use a circuit breakerHigh error rate ⇒ stop calling temporarily

Down ⇒ wait for it to come back up

Slow ⇒ gives it a chance to recover

57http://techblog.netflix.com/2012/02/fault-tolerance-in-high-volume.html

On failureReturn cached data

Return default data

Fail fast

58http://techblog.netflix.com/2012/02/fault-tolerance-in-high-volume.html

AvoidFailing

WorkerActor

Aspects + Actors

CallerDependency

InvokerAspect

DependencyStub

CircuitBreakerActor

WorkerActor

Implements circuit breaker state

machine

Equivalent of thread pool

59

@DependencyProxy annotation

@Service@DependencyProxy(circuitBreaker = "factualCircuitBreaker", timeoutInMilliseconds=750)class FactualRestaurantService extends RestaurantService { ...}

trait RestaurantService { def findNearbyRestaurants(location: Location) : Future[FindNearbyRestaurantResponse]}

60

Aspect-based Async Execution

@Aspectclass DependencyInvokingAspect {

@Pointcut("execution(* (@DependencyProxy *).*(..))") def dependencyProxyMethods {}

@Around("dependencyProxyMethods()") def invoke(jp: ProceedingJoinPoint) = { val a = AnnotationUtils.findAnnotation(jp.getTarget.getClass, classOf[DependencyProxy]) val actor = findActor(a) val timeout = Timeout(a.timeoutInMilliseconds milliseconds) actor.ask(InvokeDependency( () => jp.proceed())) (timeout) }

}

Ask actor to invoke jp.proceed() and return Future

61

See https://github.com/cer/votemeeteat

for the Actor code

62

63

AgendaThe (sometimes evil) monolithDecomposing applications into servicesHow do services communicate?Presentation layer designHow Cloud Foundry helps

Modular application

Choice of presentation layer technology

64

NodeJS is the fashionable technology

65

Why NodeJS?

66

Familiar Javascript

High-performance, scalable event-driven, non-blocking I/O model

Over 13,000 modules developed by the community

Many JavaScript client frameworks have a NodeJS counterpart, e.g. socket.io

NodeJS example

67

var http = require('http');var fs = require("fs");

http.createServer(function (req, res) { fs.readFile('somefile.txt', function (err, data) { if (err) throw err; res.writeHead(200, {'Content-Type': 'text/plain'}); res.end(data); });}).listen(1337, '127.0.0.1');

console.log('Server running at http://127.0.0.1:1337/');

Handle HTTP request

Handle file read

Socket.io - server-side

68

var express = require('express') , http = require('http') , app = express() , server = http.createServer(app) , io = require('socket.io').listen(server) ;

app.configure(function(){ app.use(express.static(__dirname + '/public'));});

server.listen(8081);

io.sockets.on('connection', function (socket) { var counter = 0; function tick() { counter = counter + 1; socket.emit('tick', counter); }; setInterval(tick, 1000);});

Serve static content from public dir

Publish event

Create a timer for socket.io client

Listen on port 8081

Socket.io - client-side

69

var socket = io.connect(location.hostname);

function ClockModel() { self.ticker = ko.observable(1); socket.on('tick', function (data) { self.ticker(data); });};

ko.applyBindings(new ClockModel());

<html><body>

The time is <span data-bind="text: ticker"></span>

<script src="/socket.io/socket.io.js"></script><script src="/knockout-2.0.0.js"></script><script src="/clock.js"></script>

</body></html>

clock.js

Connect to socket.io server

Subscribe to tick event

Bind to model

Update model

NodeJS isn’t the only game in town

70

JVM-based http://vertx.io/

A modern web application

71

Browser

Service 1

Service 2

...

HTML 5Application

Socket.ioclient

Events

RESTful WS

Server Application

Socket.ioserver

Node JS

NodeJS - using RESTful WS and AMQP

72

Node JS

Service

RabbitMQ Service

REST

AMQP AMQP

RESTRequests

Events

socket.io

73

AgendaThe (sometimes evil) monolithDecomposing applications into servicesHow do services communicate?Presentation layer designHow Cloud Foundry helps

Tomcat

Original architecture

74

Browser

WAR

MySQL Database

ShippingService

AccountingService

InventoryService

StoreFrontUI

Apache

Inventory Service Shipping Service

NodeJS

Modern architecture

75

RabbitMQ

NodeJS

Inventory Service Shipping Service

Billing Service Redis Inventory Database

Mongo Order Database

Standalone“headless” Spring/Java applications

Spring/Scala web application

MySQL Customer Database

Desktop Browser Native Mobile application HTML5 mobile application

StoreUI StoreUI StoreUI

StoreUIJavascriptAsynchronous, scalable

communication

Traditional tools: monolithic applications

76

Developing modular apps is more difficultMany more moving parts to manage•Platform services: SQL, NoSQL, RabbitMQ•Application services: your code

Who is going to setup the environments:•the developer sandbox?•...•QA environments?

77

But Cloud Foundry helps...

Applica'on  Service  Interface

Data Services

Other Services

Msg Services

Easy polyglot application deployment and service provisioning

vFabric Postgres

vFabric RabbitMQTM

Additional partners services …

OSS community

Private  Clouds  

PublicClouds

MicroClouds

Creating a platform service instance$ vmc create-service mysql --name mysql1Creating Service: OK

$ vmc services......

=========== Provisioned Services ============

+-------------+---------+| Name | Service |+-------------+---------+| mysql1 | mysql |+-------------+---------+

Multi-application manifest - part 1--- applications: inventory/target: name: inventory url: cer-inventory.chrisr.cloudfoundry.me framework: name: spring info: mem: 512M description: Java SpringSource Spring Application exec: mem: 512M instances: 1 services: si-rabbit: type: :rabbitmq si-mongo: type: :mongodb si-redis: type: :redis

80

Path to application

Required platform services

Multi-application manifest - part 2 store/target: name: store url: cer-store.chrisr.cloudfoundry.me framework: name: spring info: mem: 512M description: Java SpringSource Spring Application exec: mem: 512M instances: 1 services: si-mongo: type: :mongodb si-rabbit: type: :rabbitmq

81

Path to application

Required platform services

One command to create platform services and deploy application

$ vmc push Would you like to deploy from the current directory? [Yn]: Pushing application 'inventory'...Creating Application: OKCreating Service [si-rabbit]: OKBinding Service [si-rabbit]: OKCreating Service [si-mongo]: OKBinding Service [si-mongo]: OKCreating Service [si-redis]: OKBinding Service [si-redis]: OKUploading Application: Checking for available resources: OK Processing resources: OK Packing application: OK Uploading (12K): OK Push Status: OKStaging Application 'inventory': OK Starting Application 'inventory': OK Pushing application 'store'...Creating Application: OKBinding Service [si-mongo]: OKBinding Service [si-rabbit]: OKUploading Application: Checking for available resources: OK Processing resources: OK Packing application: OK 82

vmc push:•Reads the manifest file•Creates the required platform services•Deploys all the applications

Micro Cloud Foundry: new developer sandbox

83

Open source Platform as a Service project

App Instances Services

10.04

A PaaS packaged as a VMware Virtual Machine

Use as a developer sandbox• Use the services from Junit integration tests

• Deploy your application for functional testing

• Remote debugging from STS

Using Caldecott to tunnel into your services

84

Caldecott = TCP over HTTP

85

Your computer

Caldecott gem

Cloud Foundry

Caldecott application

Service

HTTPnative

protocol

Port NNN

Service client

nativeprotocol

Using Caldecott…$ vmc tunnel1: mysql-135e02: mysql1Which service to tunnel to?: 2Password: ********

Stopping Application: OKRedeploying tunnel application 'caldecott'.Uploading Application: Checking for available resources: OK Packing application: OK

Uploading (1K): OK Push Status: OKBinding Service [mysql1]: OKStaging Application: OK Starting Application: OK

Getting tunnel connection info: OK

Service connection info: username : uMe6Apgw00AhS password : pKcD76PcZR7GZ

name : d7cb8afb52f084f3d9bdc269e7d99ab50

Starting tunnel to mysql1 on port 10000.1: none2: mysql

Which client would you like to start?: 2

…Using CaldecottLaunching 'mysql --protocol=TCP --host=localhost --port=10000 --user=uMe6Apgw00AhS --

password=pKcD76PcZR7GZ d7cb8afb52f084f3d9bdc269e7d99ab50'

Welcome to the MySQL monitor. Commands end with ; or \g.Your MySQL connection id is 10944342Server version: 5.1.54-rel12.5 Percona Server with XtraDB (GPL), Release 12.5, Revision 188

Copyright (c) 2000, 2011, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or itsaffiliates. Other names may be trademarks of their respectiveowners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>

Running JUnit test with Caldecott

88

Configure your test code to use port + connection info

Summary

89

Monolithic applications are simple to develop and deploy

BUT have significant drawbacks

90

Apply the scale cube

91

X axis - horizontal duplicationZ ax

is - d

ata pa

rtitio

ning

Y axis - functionaldecomposition

Modular, polyglot, and scalable applicationsServices developed, deployed and scaled independently

Cloud  Provider  Interface

Applica'on  Service  Interface

Private  Clouds  

PublicClouds

MicroClouds

Data Services

Other Services

Msg Services

.js

Cloud Foundry helps

Questions?

@crichardson crichardson@vmware.comhttp://slideshare.net/chris.e.richardson/

www.cloudfoundry.com @cloudfoundry