decomposing applications for scalability and deployability (april 2012)

Decomposing applications for deployability and scalability

Chris Richardson

Author of POJOs in ActionFounder of the original CloudFoundry.com

@crichardsoncrichardson@vmware.com

1

Presentation goal

2

Decomposing applications to improve deployability and scalability

and How Cloud Foundry helps

About Chris

3

(About Chris)

4

About Chris()

5

About Chris

6

About Chris

http://www.theregister.co.uk/2009/08/19/springsource_cloud_foundry/

7

vmc push About-Chris

8

Developer Advocate for CloudFoundry.com

Signup at http://cloudfoundry.com

Promo code: CFOpenTour2012

9

Agenda§ The (sometimes evil) monolith§ Decomposing applications into services§ How do services communicate?§ Presentation layer design§ Decomposing a monolithic application§ How Cloud Foundry helps

Let’s imagine you are building an e-commerce application

10

Tomcat

Traditional web application architecture

11

Browser

WAR

MySQL Database

developtestdeployscale

ShippingService

AccountingService

InventoryService

StoreFrontUI

Simple to

Apache

But there are problems

12

Users expect a rich, dynamic and interactive experience on mobile devices and desktop

13

Java Web Application

Browser

HTTP Request

HTML/Javascript

Old style UI architecture isn’t good enough

Real-time web ≅ NodeJS

One dimensional scalability

14

§ Scales to handle transaction volumes§ Higher transaction rate required ⇒ run more instances§ Some clouds will do this automatically, others easily§ No code changes required

BUT§ Does not scale with increasing data volumes§ Caching is less effective§ Requires more memory§ Requires more I/O

Components that don’t cluster§ Most application components are easily clustered:•Most services are stateless•Server-side session state: sticky session routing/state

server

BUT there are exceptions

§ Singletons that maintain global application state:• e.g. CEP engine analyzing event stream over HTTP

§ Commercial software with a per-server license fee

§ If one application component can’t be clustered ⇒ you can’t cluster the application!

15

Obstacle to frequent deployments§ Need to redeploy everything to change one component§ Interrupts long running background (e.g. Quartz) jobs§ Increases risk of failure

Fear of change

§ Updates will happen less often§ e.g. Makes A/B testing UI really difficult 16

Large application

Slow IDESlow start times for web container

17

Scaling development

18

!=Scalable development

WAR

Shipping

Accounting

InventoryService

StoreFrontUI

§ Forces teams to synchronize development efforts§ Teams need to coordinate updates

Long-term commitment to a single technology stack§ Let’s suppose you go with the JVM•Some polyglot support•Many (but not all) JVM languages interoperate easily• e.g. Scala, Groovy modules within a Java application•But you can’t use non-JVM languages

§ Application depends on a particular:•Framework - e.g. Spring or Java EE•Container - e.g. Tomcat

§ Switching technology stack ⇒ touches entire application•Painful•Rarely done

19

20

Agenda§ The (sometimes evil) monolith§ Decomposing applications into services§ How do services communicate?§ Presentation layer design§ Decomposing a monolithic application§ How Cloud Foundry helps

3 dimensions to scaling

21

X axis - horizontal duplicationZ ax

is - d

ata pa

rtitio

ning

Y axis - functionaldecomposition

Scale by cloning

Scale

by sp

litting

simila

r thin

gsScale by splitting different things

X axis scaling - application level

22

Tomcat

Load Balancer

WAR

X axis scaling - database level

23

MySQL Master

MySQL Slave 1

MySQL Slave 2

MySQL Slave N

Writes Consistent reads

Inconsistent reads

Slaves are clones

Benefits and drawbacks of X axis splits§ Benefits•Easy and fast to implement•Handles increasing transaction volumes•Simple to manage

§ Drawbacks•Doesn’t scale with increasing data volumes•Doesn’t address application/development complexity•Couples entire application to a particular technology stack

24

Z axis scaling - data partitioning§ Each server runs the identical code base § Different servers responsible for different data§ Route request based on•Attribute of the request• Identity/attribute of the requestor/customer

§ For example:•Sharding - partition data by primary key•Free customer vs. paying customer

25

“Router”

Server 1

Server 2

...

Request:... Some attribute ...

Database server 2

Z axis scaling - database sharding

26

ID Name …

1 Ajanta

2 Eggshop

…

Partition column

RESTAURANT table

ID Name …

1 Ajanta

…

ID Name …

2 Eggshop

…

Database server 1

X axis scaling too!

Search ServiceSearch Service

Z axis scaling - application example

27

Database server 2Database server 1

Search Service Search Service

Query AggregatorUpdate Router

QueryContent

Cache Cache

Partition 1 Partition 2

Benefits and drawbacks of Z axis splits§ Benefits•Each server only deals with a subset of the data• Improved caching•Reduced memory usage and reduced I/O• Improved transaction scalability• Improved fault isolation - failure only makes part of the data

inaccessible§ Drawbacks•Doesn’t scale with increasing development and application

complexity• Increased management/deployment complexity

28

Y axis scaling - functional partitioning§ Splits monolithic application into a set of services§ Each service implements related set of functionality§ Partitioning schemes:•Partition functionality by noun or by verb•Single Responsibility Principle• e.g. Unix utilities - do one focussed thing well

29

ShippingService

AccountingService

billing web application

shipping web application

InventoryService

inventory web application

30

Y axis scaling - application level

Store Front

Store front web application

Apply X axis cloning and/or Z axis partitioning to each service

MySQL MySQL

Y axis scaling - databases

31

MySQL

Order table

....

MySQL

Customer table

....

Apply X axis cloning and/or Z axis partitioning to each database

Application

Real world examples

32

http://highscalability.com/amazon-architecture

Between 100-150 services are accessed to build a page.

http://techblog.netflix.com/

http://www.addsimplicity.com/downloads/eBaySDForum2006-11-29.pdf

http://queue.acm.org/detail.cfm?id=1394128

Benefits and drawbacks of Y axis splits§ Benefits•Scales development: focussed two pizza devops teams•Deploy services independently•Scale services independently• Improves fault isolation•Eliminates long-term commitment to a single technology

stack•Enforces well defined interfaces between components

§ Drawbacks• Implementation complexity•Deployment complexity

33

Two levels of architecture§ System level•Defines the inter-service glue: interfaces and

communication mechanisms•Slow changing

§ Service level•Defines the internal architecture of each service•Far fewer constraints on technology•Each service could use a different technology stack•Rapid evolving

34

Versus

If services are small...§ Regularly rewrite using a better technology stack§ Pick the best developers rather than best <pick a

language> developers§ Adapt system to changing requirements and better

technology without a total rewrite

35

Fred George “Developer Anarchy”

Moreover: you are not the same you ...§ Cell lifetimes:• hours - some white blood cells• days - stomach lining cells• years - bone cells• lifetime - brain cells

§ 50 to 70 billion of your cells die each day§ Yet you (the system) remains intact

36

http://dreamsongs.com/Files/WhitherSoftware.pdf

http://dreamsongs.com/Files/DesignBeyondHumanAbilitiesSimp.pdf

Can we build software systems with these

characteristics?

Too much technical debt ⇒ component death?

37

Agenda§ The (sometimes evil) monolith§ Decomposing applications into services§ How do services communicate?§ Presentation layer design§ Decomposing a monolithic application§ How Cloud Foundry helps

Inter-service communication options§ Multiple collaborating services ⇒ need a communication

mechanism§ Many choices:•Synchronous ⇔ asynchronous•Transports: HTTP, AMQP, ...•Formats: JSON, XML, Protocol Buffers, Thrift, ...•Even via the database

§ Distributed application ⇒ error handling strategies

38

ShippingService

StoreFrontUI

wgrus-store.war

AccountingService

wgrus-billing.war

wgrus-shipping.war

InventoryService

wgrus-inventory.war

MySQL

39

REST

SOAP

Thrift

...

Synchronous communication

Lots of choices

Service discovery options§ Clients need to know the coordinates of their dependencies

§ Option #1: Caller is configured with dependencies’s URL• e.g. Environment variables, system properties

§ Option #2: Services announce their location•Relies on ‘broadcast’ protocol or Gossip protocol to

announce location• e.g. RabbitMQ message broker

§ Option #3: configuration server•Maintains all the configuration information• e.g. Zookeeper

40

Benefits and drawbacks§ Benefits•Simple to implement and use

§ Drawbacks•Poor fault isolation: caller is blocked if server is down/slow•Caller needs to know server’s coordinates (URL, ...)• Less scalable than asynchronous protocols

41

StoreFrontUI

wgrus-store.war

AccountingService

wgrus-billing.war

WidgetInventoryService

wgrus-inventory.war

InventoryService

wgrus-inventory.war

MySQL

42

RabbitMQ(Message Broker)

Asynchronous message-based communication

Benefits and drawbacks§ Benefits•Decouples caller from server•Caller unaware of server’s coordinates (URL)•Message broker buffers message when server is down/slow

§ Drawbacks•Additional complexity of message broker •RPC using messaging is more complex

43

Shared databases?

44

Order Management Customer Management

Database

OrdersCustomers

Benefits and drawbacks

45

§ Benefits•Simple•Single (consistent) source of data

§ Drawbacks•Defeats the purpose of having multiple services• Less scalable•More coupling

Better: separate databases

46

Order Management Customer Management

Database

OrdersCustomers (copy)

Database

Customers

Customer Update Event

Maintaining consistency without 2PC

47

begin transaction update order tables save intent to update customer tablescommit transaction

for each saved intent begin transaction delete intent queue message to update customer commit transaction

BASE: An Acid Alternative by Dan Pritchetthttp://queue.acm.org/detail.cfm?id=1394128

dequeue messagebegin transaction if (update has not been applied) { record update as applied update customer tables } commit transaction acknowledge message

Order management

Customer management

Writing code that calls services

48

Composable futures§ Problem:•Service A needs to call services B and C and then D•Makes sense to call B and C parallel•Yet most concurrency APIs are low-level, error-prone etc

§ Solution:•Use Akka composable futures = really nice abstraction

49

val futureB = callB()val futureC = callC()

val futureD = for { b <- futureB.mapTo[SomeType] c <- futureC.mapTo[SomeType] d <- callD(b, c) } yield d

val result = Await.result(futureD, 1 second). asInstanceOf[SomeOtherType]

http://doc.akka.io/docs/akka/2.0.1/scala/futures.htmlhttp://en.wikipedia.org/wiki/Futures_and_promises

Two calls execute in parallel

And then invokes D

Get the result of D

Spring Integration

§ Builds on Spring framework

§ High-level of abstraction for building message based applications

§ Implements EAI patterns

§ Provides plumbing for exchanging messages between application components

§ Promotes loosely coupled components

§ Integrates with external messaging infrastructure: JMS, AMQP, HTTP, Email, File transfer

50

Spring Integration concepts

§ Message channel

• Virtual pipe connecting producer and consumer

§ Message endpoints

• The filter of a pipes-and-filter architecture

• Read from and/or write to channel

§ Endpoint types:

• Transformer

• Filter

• Router

• Splitter

• Aggregator

• ServiceActivator

• Inbound channel adapter - read from external source, writes to channel

• Outbound channel adapter - read from channel write to external destination

51

Spring Integration insulates components from the underlying

communication mechanism

52

Order Service

Messaging Gateway

Channel Service Activator

Shipping service

53

Example of reconfigurability - local@Servicepublic class OrderServiceImpl {

@Autowiredprivate ShippingService shippingService;

public void placeOrder() { String orderId = generateOrderId(); … shippingService.shipOrder(orderId);}

}

@Servicepublic class ShippingServiceImpl {

public void shipOrder(String orderId) { System.out.println("shipped order: " + orderId);}

}

Order Service

Messaging Gateway

Channel Service Activator

Shipping service

AMQP

RabbitMQ

AMQP Channel

54

Example of reconfigurability - distributed

Code unchanged in new deployment

Handling failure§ Errors happen (especially in distributed systems)§ Use timeouts and retries•Never wait forever•Some errors are transient so retry

§ Use per-dependency bounded thread pool with bounded queue• Limits number of outstanding requests•Fails fast if service is slow or down

§ Use circuit breaker•High rate of errors ⇒ stop calling temporarily •Avoids calling service that has issues

§ On failure•Returned cached or default data• Invoke custom error handler

55http://techblog.netflix.com/2012/02/fault-tolerance-in-high-volume.html

56

Agenda§ The (sometimes evil) monolith§ Decomposing applications into services§ How do services communicate?§ Presentation layer design§ Decomposing a monolithic application§ How Cloud Foundry helps

Modular application

Choice of presentation layer technology+

Redeploy UI frequently/independently

57

NodeJS is the fashionable technology

58

Many JavaScript client frameworks have a NodeJS counterparte.g. socket.io

NodeJS isn’t the only game in town

59

JVM-based http://vertx.io/

A modern web application

60

Browser

Service 1

Service 2

...

HTML 5Application

Socket.ioclient

Events

RESTful WS

Server Application

Socket.ioserver

Node JS

Alternatively: multiple front-end servers

61

Browser

Node JS Service 1

Service 2

...

HTML 5Application

Node JS

Node JS

Needs single-sign-on

NodeJS - using RESTful WS and AMQP

62

Node JS

Service

RabbitMQ Service

REST

AMQP AMQP

Requests

Eventssocket.io

Updating the UI is easy

63

§ Update the UI independently of rest of system§ Easily run A/B tests § Enables fast iteration of the UI

http://theleanstartup.com/principles

But coordination with backend changes required§ Let’s imagine that you are deploying an advanced search

feature:•Enhancements to search service•Enhancements to UI

§ Before•Deploy new war

§ Now:•Some coordination required•Deploy updated backend service•Deploy updated NodeJS and browser code•Enable feature using feature switch

64

65

Agenda§ The (sometimes evil) monolith§ Decomposing applications into services§ How do services communicate?§ Presentation layer design§ Decomposing a monolithic application§ How Cloud Foundry helps

The challenge§ In the beginning:• It would be easy to use Y axis/functional decomposition•But you don’t need it and it would slow you down

§ Later on:•You do need it•But your application logic is tangled and decomposition is

difficult

66

Lower cost way to start with Y axis decomposition§ Enforce boundaries between services, e.g. DTOs§ But run in a single JVM with a single database§ Use Spring Integration•Course-grained message-based communication•Via in-memory channels

§ Downside:• Implementing the glue layers are extra work

67

Application

Special case: Components that don’t cluster

68

Monitoring

Esper

Rest of domain classes

AC

D

B

Doesn’t cluster ⇒ No X axis scaling/cloning

Tangled dependencies prevent Y axis scaling

Don’t let this happen!

Use well-defined interfaces

Ensure loose coupling

69

Monolithic application

Traditional application architecture

70

Service A Service B Service C

Shared domain model

A C

B

X Z

Y

Difficult to untangle

Service

Domain model 2

Application

Extracting a service = untangling dependencies

71

Service A Service BService C

Domain model 1

A C

B

X

Z

Y

Trouble!

Untangling dependencies

72

§ Domain object is part of an association•Read only - replicate•Update only - notification interface•Update then read - replicate

§ Domain object is a request parameter•Read-only - replace with DTO•Update only - notification interface•Update then read - tricky

§ Note - Need acyclic dependencies• If used object isn’t part of a service = assumption is that it

belongs in calling service

class A { private B other;}

class A { void m(B other) { ... }}

Bounded context is a useful idea§ Different parts of a system ⇒ different domain models§ Different services have a different view of some domain

objects, e.g.•User Management = complex view of user•Rest of application = PK + ACL + Name

§ Different services can have a different domain model§ Services exchange messages to synchronize data

73

Domain model 2

Untangling dependencies 1

74

Domain model 1

A C

B

X

Z

Y

doSomething(Y)

getSomething()

doSomethingElse()

Service A Service B Service C

Domain model 2

Untangling dependencies 2

75

Domain model 1

A C

B

X

Z

Y

doSomething(YDTO)

doSomethingElse()

Facade A

Facade B

Service A Service B Service C

Domain model 2

Untangling dependencies 3

76

Domain model 1

A

C

B

X

Z

Y

doSomething(YDTO)

doSomethingElse()

Facade A

Facade B

Message Broker Listener

Listener

Service A Service B Service C

77

Agenda§ The (sometimes evil) monolith§ Decomposing applications into services§ How do services communicate?§ Presentation layer design§ Decomposing a monolithic application§ How Cloud Foundry helps

Traditional tools: monolithic applications

78

Developing service-oriented apps is harder§ Many more moving parts to manage• Infrastructure services: SQL, NoSQL, RabbitMQ•Application services

§ Who is going to setup the environments:• the developer sandbox?• ...•QA environments?

79

Cloud Foundry makes it easier§ Easy to deploy applications§ Easy to provision services§ Manifest describes the structure of a set of services§ vmc env-add for binding 'app services' together§ Micro Cloud Foundry is your personal cloud/sandbox§ Caldecott exposes services for use by integration tests

80

Services available on Cloud Foundry

81

Creating a service instance$ vmc create-service mysql --name mysql1Creating Service: OK

$ vmc services......

=========== Provisioned Services ============

+-------------+---------+| Name | Service |+-------------+---------+| mysql1 | mysql |+-------------+---------+

Multi-application manifest - part 1--- applications: inventory/target: name: inventory url: cer-inventory.chrisr.cloudfoundry.me framework: name: spring info: mem: 512M description: Java SpringSource Spring Application exec: mem: 512M instances: 1 services: si-rabbit: type: :rabbitmq si-mongo: type: :mongodb si-redis: type: :redis

83

Path to application

Required services

Multi-application manifest - part 2 store/target: name: store url: cer-store.chrisr.cloudfoundry.me framework: name: spring info: mem: 512M description: Java SpringSource Spring Application exec: mem: 512M instances: 1 services: si-mongo: type: :mongodb si-rabbit: type: :rabbitmq

84

Path to application

Required services

One command to create services and deploy application$ vmc push Would you like to deploy from the current directory? [Yn]: Pushing application 'inventory'...Creating Application: OKCreating Service [si-rabbit]: OKBinding Service [si-rabbit]: OKCreating Service [si-mongo]: OKBinding Service [si-mongo]: OKCreating Service [si-redis]: OKBinding Service [si-redis]: OKUploading Application: Checking for available resources: OK Processing resources: OK Packing application: OK Uploading (12K): OK Push Status: OKStaging Application 'inventory': OK Starting Application 'inventory': OK Pushing application 'store'...Creating Application: OKBinding Service [si-mongo]: OKBinding Service [si-rabbit]: OKUploading Application: Checking for available resources: OK Processing resources: OK Packing application: OK Uploading (5K): OK Push Status: OKStaging Application 'store': OK Starting Application 'store': ... 85

vmc push:•Reads the manifest file•Creates the required services•Deploys all the applications

Application configuration via environment variables

String value = System.getenv("PAYMENT_SVC")

@Value("#{systemEnvironment['PAYMENT_SVC']}")private String envVariable;

$ vmc env-add cf1 PAYMENT_SVC=http://...Adding Environment Variable

[PAYMENT_SVC=http://...]: OKStopping Application: OKStaging Application: OK

Micro Cloud Foundry: new developer sandbox

87

Open source Platform as a Service project

App Instances Services

10.04

A PaaS packaged as a VMware Virtual Machine

Use as a developer sandbox

• Use the services from Junit integration tests

• Deploy your application for functional testing

• Remote debugging from STS

Using Caldecott…$ vmc tunnel1: mysql-135e02: mysql1Which service to tunnel to?: 2Password: ********Stopping Application: OKRedeploying tunnel application 'caldecott'.Uploading Application: Checking for available resources: OK Packing application: OK Uploading (1K): OK Push Status: OKBinding Service [mysql1]: OKStaging Application: OK Starting Application: OK Getting tunnel connection info: OK

Service connection info: username : uMe6Apgw00AhS password : pKcD76PcZR7GZ name : d7cb8afb52f084f3d9bdc269e7d99ab50

Starting tunnel to mysql1 on port 10000.1: none2: mysqlWhich client would you like to start?: 2

…Using CaldecottLaunching 'mysql --protocol=TCP --host=localhost --port=10000 --

user=uMe6Apgw00AhS --password=pKcD76PcZR7GZ d7cb8afb52f084f3d9bdc269e7d99ab50'

Welcome to the MySQL monitor. Commands end with ; or \g.Your MySQL connection id is 10944342Server version: 5.1.54-rel12.5 Percona Server with XtraDB (GPL),

Release 12.5, Revision 188

Copyright (c) 2000, 2011, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or itsaffiliates. Other names may be trademarks of their respectiveowners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>

Running JUnit test with Caldecott

90

Configure your test code to use port + connection info

Summary§ Monolithic applications are simple to develop and deploy

BUT applying the scale cube

§ Decomposes your application into services§ Enables scaling for transactions and data volumes§ Tackles application complexity§ Enables scaling for development§ Enables frequent, independent deployments§ Make it easy to leverage other technologies

AND

§ Cloud Foundry simplifies the development and deployment of “service-oriented” applications

91

Questions?

@crichardson crichardson@vmware.com

www.cloudfoundry.com promo code CFOpenTour2012