xa transaction & integration use case with openesb

XA transaction & integrationUse case with OpenESB

Agenda

Pymma Consulting: Services versioning with WSDL 2

XA for an ideal worldXA and the real worldXA and IntegrationOpenESB featuresConclusions

THE IDEAL WORLD


The ideal world


Process

Context N

Context N

Partner 1

Context N

Partner 2

Context N

Partner 3

Before the process starts

The ideal world


Process

Context N+5

Context N+3

Partner 1

Context N+2

Partner 2

Context N +7

Partner 3

During the process

The ideal world


Process

Context N+8

Context N+5

Partner 1

Context N+6

Partner 2

Context N +9

Partner 3

An error occurs during the process

The ideal world


Process

Context N+8

Context N+5

Partner 1

Context N+6

Partner 2

Context N +9

Partner 3

Partners are informed about the error

The ideal world


Process

Context N

Context N

Partner 1

Context N

Partner 2

Context N

Partner 3

Partners and process rollback to the original contexts

The ideal world


Process

Context N

Context N

Partner 1

Context N

Partner 2

Context N

Partner 3

Using XA for transaction and recovery

XA

XACommitRollback

CommitRollback

CommitRollback

XA

The ideal worldXA is perfect for our ideal world We are all familiar with XA or two phases commit transaction– Mainframe, Database, Transaction manager, J2EEserver

XA Transactions provide – Once and only once delivery /consistency

XA (2 phases commit) has been the basis of 70’s 80’s and 90’s IT softwares


THE REAL WORLD


The ideal worldXA is a concept for an care bear’s worldThe ideal world is difficult to implementXA needs special conditions for working well– Uniformity / Proximity

Uniformity – Does not work in a non uniform environment– Not implemented with heterogeneous applications

Proximity– XA works only when partners collocated – Being on the same network is not a collocation

In the 00’s, Internet blasted XA and the ideal world


Internet blasted the ideal world

From stand alone to multi-partners


Silo Application

Module

Module

Module

Collaborate

Providerclient

Partner

Partner

Client

Provider Partner

Partner

Client

Provider


From one to multiple technologies


IBM Mainframe

Unix

Mainframe

Linux

WindowsWeb

Software PackageExternal

Partners

FTP

Http

JMS

NFS

TCP/IP

Java

.Net

C++

Cobol


Web applications need more power than ever


Up to 10k users

Up to 100 M users


Web applications need more power than ever


Server

ServerServer

ServerServer

ServerServer

ServerServer

ServerServer

The Real worldIn the 00’s – Many projects failed in distributed environment– Business & management worried about this fact

• Why IT could not provide what it did in the 70’s 80’ and 90’s ?

High management said:– Difficulties to implement XA came from

• A lack of knowledge from IT people• Bad technologies and platforms choices

Moreover in the 00’s big editors lied– Saying their tools support distributed XA– Helped by famous analysts

Let’s explain why XA was so difficult to implementPymma Consulting: Services versioning with WSDL 17

The real world


Client /Server Architecture

SQL

18

Centralised Architecture

The real world

The network is not a simple extension – Latency creates partitioning– One group can work an not the other


Latency

M a n y a r c h i t e c t s t h o u g h t t h a t d i s t r i b u t e d a r c h i t e c t u r e s b e h a v e l i k e c e n t r a l i z e d a r c h i t e c t u r e s .

The real world

You create partitions – When you work with many platforms – When you use different technologies – When you create a geographical distribution

Partition means:– A part of your system can work when one is stopped


Partition

The real world and CAP theorem


In 2002 the CAP theorem saved us– CAP stands for • Consistency Availability and Partitioning

– Set out by Eric Brewer • Mathematically proved by S. Gilbert et N. Lynch (MIT)

CAP is a mathematical answer to – The 00’s management and Editors

Let’s see what it says



CAP Theorem (Consistency, Availability, Partition)– We need 3 properties needed for a network: consistency, availability

and partitioning. At most, it is possible to get two of them.

C A

P

• Consistency + Availability • Single site / Cluster database• Two phase commit

C AP

• Consistency + Partition • Distributed database / Distributed looking• Pessimistic looking

C AP

• Availability + Partition • Distributed cache/ DNS• Optimistic looking



office

officeoffice


First Case I want consistency


Office Brussels

Office Berlin

Office Nice

Inform

Inform

Lock

C AP


First Case I want consistency


Office Brussels

Office Berlin

Office Nice

Get Data

Data not available

C AP

Data not available


First Case, we chose consistency


Office Brussels

Office Berlin

Office Nice

Inform

Inform

Unlock

C AP

Data are consistent but not always available


Second case I want availability


Office Brussels

Office Berlin

Office Nice

Get data

C AP




Office Brussels

Office Berlin

Office Nice

process data

C AP

Get data

process data




Office Brussels

Office Berlin

Office Nice

C AP

Put dataPut data

Data are availablebut not always consistence



CAP Theorem (Consequences)– In a partitioned environment

• We cannot get at the same time consistency and availability• Mathematically demonstrated that XA is not available• We must choose which property to promote

– Ex : statistics data/ Historic data / transfer / Customer account

Why is the CAP theorem is new for us? – Limited latency in the local area network– Database editor propaganda – Not teach in the universities yet – Many architects ignore CAP theorem existence – Similitude with the uncertainly principle of Heisenberg



Concept difficult to admit: Impact on mentality and habit– Business people are use to design with the concept of

transaction– B.As write specifications relying on XA transactions – Going out transactions seems a regression for the business– Most of architects are comfortably relying on transactions

to provide consistency– They don’t know how to do without it– Some developers cannot live without “Commit and

Rollback”

The Real world (summary)In the 00’s IT world changed– Internet destroyed the ideal world

It introduce biodiversity in IT systems– Many partners (some not XA)– Multiple technologies (some not XA)– Many protocols (some not XA)

XA does not work in a non uniform environmentXA doesn’t exist in the real worldXA doesn’t exist in real life


INTEGRATION AND XA


Integration and XA


All together Manual Validation

Receive Transfer order by email

Check rights

Swift transfer

Send email for confirmation

Check account

Banker validation

Log transaction


Integration and XA

Integration means– Multiple partners– Multiple Technologies– Multiple protocols

Typically an heterogeneous environment– Impossible to apply XA

For these environment could we provide:– Guarantee of delivery – Consistency

Question: – Can integration stand with strong consistency and high

guarantee of delivery?

OPENESB FEATURES: CONSISTENCY & GUARANTEE OF DELIVERY


OpenESB features

OpenESB is an integration tool– Relying on JBI specifications (Java Business Integration)

OpenESB proposes natively advanced features – High availability – Quality of services – Consistency

We will review these featuresThey are used by architects & designers to increase quality


Pymma notation


BindingComponent

Binding Component XA implementation

Service without side effects

Service without side effects

XA Implementation

Service with side effects

Service with side effects

XA Implementation

Pymma notation


BPEL

Catch

Compensation

The process is atomic

The process is persistent

Regular process

When an error occurs

Compensation part of the

process

N

RedeliveryN times

Integration and business processes


Short processesFew partnersAutomatic tasks

Emitter

Receiver

Integration process

BPEL

Few seconds

max

Automatic tasks

Integration process

OpenESB Technical concepts


Longer processesMany partnersAutomatic & Manual tasks

Emitter

Receiver

Business process

BPEL

Fromsecondsto weeks

Automatic tasks

Manual tasks

Business process

Service acknowledgement


partner partner

JBI environment

Acknowledgement

Acknowledgment is required in a JBI environment

Emitter Receiver

Acknowledgement with BPEL

ACKBPEL

ACK

BPEL Atomic property


Emitter

Receiver

ACK is sent just after the reception: Atomic = false

ACK

BPEL

Receive

Activity

Activity

invoke

BPEL Atomic property


Emitter

Receiver

No ACK is sent when a non catch error occurs in the process : atomic = true

ACK

BPEL

Receive

Activity

Activity

invoke

XX

Side effect and compensation


Emitter

Receiver

process

Simple process

A

B



Emitter

Receiver

process

Simple process with side effect

Side effect

A

B

Catch



Emitter

Receiver

process

Simple transactional process with side effect

A

B

There is no RollbackThere is no Rollback



Compensation

A

Side effects

C: AService A

Compensation

Increases side effects

Suppresses or decreasesside effects

Service A



Insert in database

Delete in database

Compensationcomplete

Write in audit file

Write in audit file

Compensationpartial

Put message in Topic

Put message in Topic

CompensationPartial

Add number in a total

Subtract number in

total

Compensationcomplete



CService A

CService B

CService C

CService D

Service A Service B Service C Service D

Normal process

Compensation process

CService A

CService B

CService C

CService D

Service A Service B Service C Service D



Save a record

C: delete the record

DB

Insert into ....Audit service.

V1

Delete from ....SQL

Save a record


Write in file

File

Audit service. V2

Delete from ....SQL



Save a record


DB

Insert into ....

Audit serviceV1

Delete from ....

Save a record


Write in file

Audit serviceV2

Delete in file

File

F-1(x)

F-1(x)

F(x)

F(x)

Qualities & Redelivery Extension


Redelivery Extension– Max attempts N times– Wait time – On failure • Delete / error / redirect / suspend

BPEL

BPEL

N

N

N

N

N

NBPEL



BPEL Http

Non reliable connection3

Wait 2s

External Service

BPEL JMS MQ2

Redirect

Http Web Service

2

Error

FTP FTP ServerSuspend

BPEL2

Monitoring

Try to fix the problem

Persistence Principle


C C CC

Side effect

No transactional process

After a failure

Full transactional process

RollbackAfter a failure



C C CC

BPEL internal context

DB

C C CC


DB



Limits side effect Increases guaranty of deliveryDecreases the performances– Powerful DB server can be used– Cache distributed instead DB

Solutions can be found to use efficiently persistence

C C CC


DB

High availability


Machine 3

Machine 2

Machine 1

Domain

Cluster

Cluster

Instance

Instance

ESB

ESB

Instance

Instance

ESB

ESB

Instance

Instance

ESB

ESB

App.

App.

App.

App.

App.

App.

App.

App.

Node agent

Node agent

Node agent

Cluster

High Availability

OpenESB has been designed for multi-instance– Native design – Don’t use containers cluster features

It provides easily high Availability Architects uses this features– To increase Quality of services– Useful to secure processes and services

Must be associated with good BP designs


HOW TO USE OPENESB FEATURES


Cleverness and pragmatismThere are 3 delivery types– 0,n / 1,n / 1,1 delivery

There is not system that can really provide 1,1– XA system are close from it – OpenESB can provide 1,n in a non XA environment– Many native OpenESB features to achieve 1,n delivery

Pragmatism – Do not try to reach 1,1– increase business and management awareness about it

Cleverness– Identify what is really important, exceptional, vital– Don’t focus on the exceptional– Don’t forget that human is the best to process exception


Cleverness and pragmatismCleverness start with BP design– Avoid stateful processes promote stateless

• Use hydration and dehydration

– Avoid Multipart conversation – Plan entry features to avoid doubloon– Plan compensation messages at the design level

Be pragmatic– Cancelation message is often better that transaction– Human is very smart to solve complex issue

• Recording vital data in a data is critical• Process it in a simple way• Process automatically the most common errors• Human or external process to solve complex error


80% -20%

Cleverness and pragmatism

Spend time on happy ways and simple errors– Design and processing are straightforward– Simple error can be process easily

• QOS / Redelivery • Compensation • Atomicity

– That’s architect and designer job

Don’t spend time on exceptional– Don’t try to replace (or replay) XA– What happens if a meteorite fails on my computer– You will increase development and budget– increase maintenance complexity


Cleverness and pragmatism

Does this philosophy works– Google, Ebay, Amazon follows this philosophy– They cancelled XA for scalability or Consistency

Pymma helps its customers to follow this way – Large to very large projects > many K men days– KBC Bank, Leroy Merlin, Kiabi, IFDS– They all need reliability and consistency – But scalability and efficiency for integration

They found in OpenESB a solution We are sure you will find in OpenESB your solution


QUESTIONS AND COMMENTS


THANK YOU

xa transaction & integration use case with openesb

Documents