Enterprise Integration and Service-Oriented Design

Hossam Karim

Agenda The Enterprise Integration Problem Enterprise Integration Principles, Styles and

Messaging Concepts Service-Oriented Design Approach Frameworks and Implementations References Conclusion

The Enterprise Integration Problem

Enterprise integration requires a significant shift in corporate politics

Integration efforts typically have far-reaching implications on the business

The lack of widely implemented standards addressing core integration problems

The concept and technology learning curve

Enterprise Integration Principles

Integration is connecting computer systems, companies or people

Enterprise integration is the task of making disparate applications work together to produce a unified set of functionality

Integration Types

Information Portals Data Replication Shared Business Function Service-Oriented Architecture Distributed Business Process Business to Business Integration

Enterprise Integration TypesInformation Portals

Reso

urce

A

Reso

urce

B

Reso

urce

C

Basic data aggregation Single source of information

P o r t al

Enterprise Integration TypesData Replication

Data Store A

Data A

cces

s

Applic

ation

B

Data Store B

Data A

cces

s

Data moves on the database tier Consistency is dependent on the DBMS

implementation

Applic

ation

A

Enterprise Integration TypesShared Business Function

Applic

ation

A

Logic

A

Applic

ation

B

Logic

B

Shar

ed

Func

tion

Moves the control to the shared function logic Tightly couples all clients to the application

interfaces

Enterprise Integration TypesService-Oriented Architecture

Serv

ice

A Serv

ice

B Serv

ice

C Serv

ice

D

Applic

ation

Se

rvice

Applic

ation

Connects distributed applications and exposes services through a standard contract

Complex to design and implement

Enterprise Integration TypesDistributed Business Process

Busin

ess

Proc

ess

Func

tion A

Func

tion B

Func

tion C

Single point of invocation Requires other integration solutions to

function

Enterprise Integration TypesBusiness-To-Business Integration

Func

tion A

Func

tion B

Higher level of integration Still requires an integration solution

Enterprise Integration Styles

File Transfer Shared Database Remote Procedure Invocation Messaging

Enterprise Integration StylesFile Transfer

Applic

ation

A

Applic

ation

B

ExportImportSh

ared

Dat

a

Simplest form of integration Lacks timeliness and scalability

Enterprise Integration StylesShared Database

Appl

icat

ion

A

Data Store

Appl

icat

ion

B

Appl

icat

ion

C

Data A

cces

s

Data A

cces

s

Data A

cces

s

Promotes consistency Lacks portability and scalability

Enterprise Integration StylesRemote Procedure Invocation

Applic

ation

A

Applic

ation

B

StubSkeleton

Function Call

Reply

Provides more than data sharing Tightly couples integrated parties

Enterprise Integration StylesMessaging

Applic

ation

A

Applic

ation

A

Applic

ation

A

Event Layer

Message Bus

Transfer packets of data immediately, reliably, and asynchronously, using customizable formats.

Tightly Coupled Interfaces

APIA

APIC

APIB

DataA

Application A Application CApplication B

DataB

DataC

Requires minimum n(n-1)/2 interfaces, where n is the number of integrated applications

Loosely Coupled Interfaces

Requires exactly n interfaces, where n is the number of integrated applications

MessagingInterface A

MessagingInterface C

MessagingInterface B

DataA

Application A Application CApplication B

DataB

DataC

Messaging Bus

Enterprise Integration Using Messaging

Construction Channels Endpoints Routing Transformation Management Messaging Models Transactions

Message Construction

A message is a formatted unit of data A message consists of a header, properties and a body A sender marshals the data to construct the message A receiver un-marshals the data into its own data model

Message ReceiverSender

Message ConstructionExample: Spring Integration

Message

++++

getId ()getHeader ()getPayload ()isExpired ()

: Object: MessageHeader: T: boolean

<T>

Correlation ID

Pure Java object as a message body

Simplest form of a modeled message Uses generics to specialize message body type No marshaling or un-marshaling needed

Generic message body type

Header support

Message ConstructionExample: Java Business Integration

NormalizedMessage

+++++++++++

addAttachment (String id, DataHandler content)getContent ()getAttachment (String id)getAttachmentNames ()removeAttachment (String id)setContent (Source content)setProperty (String name, Object value)setSecuritySubject (Subject subject)getPropertyNames ()getProperty (String name)getSecuritySubject ()

: void: Source: DataHandler: Set: void: void: void: void: Set: Object: Subject

XML bodyAttachments support

Merged Properties andHeader supportSecurity support

Body is XML based Supports attachments and security subject

Message ConstructionExample: Apache Camel

Message

+++++++++++++++

getMessageId ()setMessageId (String messageId)getExchange ()getHeader (String name)getHeader (String name, Class<T> type)setHeader (String name, Object value)removeHeader (String name)getHeaders ()setHeaders (Map<String, Object> headers)getBody ()getBody (Class<T> type)setBody (Object body)setBody (Object body, Class<T> type)copy ()copyFrom (Message message)

: String: void: Exchange: Object: T: void: Object: Map<String, Object>: void: Object: T: void: void: Message: void

Correlation ID

What is this?

Pure POJO body

Header support

Message ConstructionMessage Exchange Patterns

<<Enum>>

ExchangePattern

++++++++...

InOnlyRobustInOnlyInOutInOptionalOutOutOnlyRobustOutOnlyOutInOutOptionalIn...

: EnumConstant: EnumConstant: EnumConstant: EnumConstant: EnumConstant: EnumConstant: EnumConstant: EnumConstant...

+++++- <<staticInitializer>>

getWsdlUri ()isInCapable ()isOutCapable ()isFaultCapable ()fromWsdlUri (String wsdlUri)_STATIC_INITIALIZER ()...

: String: boolean: boolean: boolean: ExchangePattern: void

Exchange Patterns

WSDL 2.0 based message exchange patterns

Exchange

++++++++++++++...

getPattern ()getProperty (String name)getProperty (String name, Class<T> type)removeProperty (String name)getProperties ()getIn ()getOut ()getOut (boolean lazyCreate)getFault ()getFault (boolean lazyCreate)isFailed ()copy ()copyFrom (Exchange source)getExchangeId ()...

: ExchangePattern: Object: T: Object: Map<String, Object>: Message: Message: Message: Message: Message: boolean: Exchange: void: String...

Message ConstructionMessage Exchange

Messages are encapsulated into a message exchange The pattern identifies what is encapsulated

Actual Message

This exchange pattern

Messaging Channels

A channel is a gateway for transmitting messages from a sender to a receiver

Sender and receiver do not know about each other Can be referred to as a pipe

SenderApplication

ReceiverApplication

MessageChannel

Message ChannelExample: Spring Integration

MessageChannel

+ DEFAULT_CAPACITY : int = 100

+++++++++

getName ()setName (String name)getDispatcherPolicy ()send (Message message)send (Message message, long timeout)receive ()receive (long timeout)clear ()purge (MessageSelector selector)

: String: void: DispatcherPolicy: boolean: boolean: Message: Message: List<Message<?>>: List<Message<?>>

Pure POJO model

Message ChannelExample: Java Business Integration

DeliveryChannel

++++++++++

close ()createExchangeFactory ()createExchangeFactory (QName interfaceName)createExchangeFactoryForService (QName serviceName)createExchangeFactory (ServiceEndpoint endpoint)accept ()accept (long timeout)send (MessageExchange exchange)sendSync (MessageExchange exchange)sendSync (MessageExchange exchange, long timeout)

: void: MessageExchangeFactory: MessageExchangeFactory: MessageExchangeFactory: MessageExchangeFactory: MessageExchange: MessageExchange: void: boolean: boolean

Factory method pattern, creates message exchange factories

Supports synchronous and asynchronous exchanges

Messaging Endpoints

An endpoint is a client of a messaging application that is capable of sending and receiving messages

The Message Endpoint encapsulates the messaging system from the rest of the application and customizes a general messaging API for a specific application and task

Data DataEndpoint EndpointChannelMessage

SenderApplication

ReceiverApplication

Message EndpointExample: Spring Integration

MessageHandler(handler)

+ handle (Message<?> message) : Message<?>

MessageEndpoint

+++

getName ()getSubscription ()getConcurrencyPolicy ()

: String: Subscription: ConcurrencyPolicy

<<AnnotationType>>

MessageEndpoint

+++

inputdefaultOutputpollPeriod

: String: String: int

= "" = "" = 0

<<AnnotationType>>

Handler

Message EndpointExample: Spring Integration

Message Processing Problem

DataEndpointChannel

ReceiverApplication

(n) messages

Exhausted at (n - x) messages

Exhausted at (n - y) messages

Client sends (n) messages Channel’s capacity is (n-x) messages Receiver application exhausted at (n-y) messages

Message Processing ProblemSEDA: Staged Event-Driven Architecture

Event Handler

Thread Pool

Event Queue

Controller

SEDA Stage

The Event Queue accepts incoming events

The Event Handler handles incoming events

The Thread Pool provides threading capabilities to the event handler

The Controller adjusts resources allocation and scheduling dynamically

Message Processing ProblemSEDA: Staged Event-Driven Architecture

IncomingMessages

ReceiverStage

Stage A

Stage B

Stage C

OutgoingMessages

OutgoingMessages

OutgoingMessages

Message Routing

A Message Router consumes a Message from one Message Channel and republishes it to a different Message Channel, depending on a set of conditions

Message InputChannel Message

Router

OutputChannel

A

OutputChannel

B

ReceiverA

ReceiverB

Message RouterExample: Spring Integration

Message Transformation

A Message Translator is a filter that translates one data format to another

IncomingMessage

TranslatedMessage

Translator

Message FlowExample: The VETRO Pattern

Valid

ate

Route

Opera

te

Transf

orm

Enric

h

Message 1 2 3 4 Message 5

All numbers and kinds of filters can be applied to a message before processing

System Management

The Control Bus uses the same messaging mechanism used by the application data but uses separate channels to transmit data that is relevant to the management of components involved in the message flow

Processor A

Processor B

Processor C

Message Flow

Control Bus

Configuration

Heartbeat

Test Messages

Exceptions

Statistics

Live Console

Message X Message X’

Messaging Models

Producer(Publisher)

Topic

Consumer

Consumer

Consumer

Each subscriber (zero or more)Gets a copy

Publish and subscribe ( 1 → Many)

Producer(Queue Sender)

Queue

Consumer

Consumer

Consumer

Potential Receivers (zero or one)

Point-to-Point ( 1 → 1)

Publish/Subscribe Model Point-to-Point Model

Store and Forward Messaging

Guaranteed once and only once message delivery Transparent to the messaging clients

Producer 1. sendTopic

Queue

Data Store

2. store

Consumer3. forward

Transactional MessagingLocal Transactions

Producer ConsumerMessaging

Server

1. send

2. send

3. commit

4. receive

5. receive

6. commit

Producer sends one or more messages in a transactional context Producer commits the transaction Consumer receives all messages Consumer commits the transaction

Transactional MessagingDistributed Transactions

Producer sends one or more messages in the transaction manager context

Producer updates another resource Producer commits the transaction

ProducerMessaging

Server

1. send

Data Store

2. Update

3. Commit, Rollback or Recover

XA Transaction Manager

Messaging Protocol Stack

TCP/IP

HTTP/S

TCP/IP TCP/IP TCP/IP

HTTP/S HTTP/S

TCP/IP

SSL

JMS

XML

MessagingBus

MessagePayload

JMSJMS

SOAP

MessagingBus

MessagePayload

XML

MessagingBus

MessagePayload

SOAP

MessagingBus

MessagePayload

MessagingBus

MessagePayload

NetworkingProtocol

TransportProtocol

MessagingProtocol

PayloadProtocol

Clients expect simple and widely used protocols Legacy applications communicate in proprietary protocols Firewalls require tunneling through opened ports

Service Meta-Model

1..*operations

0..*infault

0..*outfault

1..1pattern

0..*input

0..*output

1..1message

1..*endpoints

1..1binding

Interface

+ name : String

Binding

++

nametype

: String: URI

Service

+ name : String

Endpoint

++

nameaddress

: String: URI

Operation

+ name : String

Fault

Message

+ label : String

<<enumeration>>

MessageExchangePattern

+ name : String

Service Component Model

ServiceComponent

<<delegate>><<delegate>>

ServiceImplementation

+ <<InOut>> businessOperation (Message input) : Message

http-endpoint jms-endpoint

<<remote>>BusinessInterface : 1

<<remote>>BusinessInterface : 2

Exposed Interface

Protocol SpecificEndpoint

ServiceImplementation

EncapsulatingComponent

Service Component Clients

ServiceComponent

<<delegate>><<delegate>>

ServiceImplementation

+ <<InOut>> businessOperation (Message input) : Message

http-endpoint jms-endpoint

<<remote>>BusinessInterface : 1

<<remote>>BusinessInterface : 2

<<client>>HttpClient

http

<<client>>JmsClient

jms

Interface Binding

Protocol Dependent

Client

Service-Oriented Design Approach

Message Bus

Service-OrientedEnvironment

Com

ponent

Service Endpoints

Service Transport ProtocolSupport

ServiceContract

Messaging BusEndpoint

NormalizedMessages

Service-Oriented Design Approach Example: Java Business Integration

JBI Runtime Environment

Service Unit

Service Assembly

Service Unit Service Unit

Jbi.xml Jbi.xml Jbi.xml

Jbi.xml

Jbi.xml

Component Shared Library

Service Engine

Binding Component

Jbi.xml

Normalized Message Router

Install

Deploy

Message Bus

ActivateEndpoints

CommonLogic

ExposeServices

Service-Oriented Design Approach Example: Java Business Integration

Serv

ice

Engin

e

Directory Server

Relational Database

XML Native Database

Bindin

g

Com

ponent

Service Instances Services are exposed on arbitrary

supported protocols Messages are normalized and

delivered to their channels Messages are routed through the

normalized message router to the target service engine

The service engine performs its task and replies to the component if appropriate

Frameworks and Implementations Spring Integration

http://www.springframework.org/spring-integration/ Apache ServiceMix

http://servicemix.apache.org/ Apache Camel

http://activemq.apache.org/camel/ Sun OpenESB

https://open-esb.dev.java.net/ Mule

http://mule.mulesource.org/

References Enterprise Integration Patterns: Designing, Building, and

Deploying Messaging Solutions - Gregor Hohpe, Bobby Woolf

Enterprise Service Bus – Dave Chappell SOA: Principles of Service Design - Thomas Erl SEDA: An Architecture for Well Conditioned, Scalable

Internet Services - Matt Welsh, David Culler, and Eric Brewer Java™ Business Integration (JBI) 1.0 – JSR 208 Specification -

Ron Ten-Hove, Peter Walker

Thanks to Dave Chappell for granting permission to use his diagramming notation in this presentation

Conclusion Don’t reinvent the wheel Keep close to the standards Spend time understanding and evaluating your

infrastructure Not all frameworks and design solutions are suitable for

each environment Design to scale, but don’t over design Explore integration solutions and frameworks Practice service-oriented design and implementation