coherence in an event driven architecture

Coherence in an Event Driven Architecture

24 Oct 2013

1Friday, 25 October 2013

About Intelligent Pathways

• Australian technology company founded in 2003• Specialising in enterprise integration and application

development• Our approach is aligning technology with business

objectives• Strong track record of delivery• In depth knowledge of Oracle Fusion Middleware; early

adopters of SOA


The Customer

Passenger

Book Flights And

Ancillary Products

Airport Transfers

Process Payments

Issue Travel

Documents

Operate Lounges

Baggage Services

Operate Flights

In-flight Services

Check-in and

Boarding Control


Enterprise functions to serve customerFunctions span business unitsBased on passenger journey

The Customer Data

PNR

Book Flights And

Ancillary Products

Airport Transfers

Process Payments

Issue Travel

Documents

Operate Lounges

Baggage Services

Operate Flights

In-flight Services

Check-in and

Boarding Control


Passenger represented as a PNR - Passenger Name RecordBusiness functions update state on PNRPNR contains huge amounts of data on customer

The Reservation Platform

Amadeus

Sell (RES)

Fly (CM)

Book Flights And

Ancillary Products

Airport Transfers

Process Payments

Issue Travel

Documents

Operate Lounges

Baggage Services

Operate Flights

In-flight Services

Check-in and

Boarding Control


PNR source of truth is the reservation platformGreat opportunity to mine the PNR for events for the journey of the passengerSell = Booking and TicketingFly = Customer Journey

PNR Events

Amadeus

Sell (RES)

Fly (CM)

Book Flights And

Ancillary Products

Airport Transfers

Process Payments

Issue Travel

Documents

Operate Lounges

Baggage Services

Operate Flights

In-flight Services

Check-in and

Boarding Control

PNR Events


The PNR Feed from Amadeus contains PNR messages based on changes to the PNRData changes trigger a new message Message contains current state of the PNR, the previous state of the PNR and a report on what changed

Event Processing ConceptsEvent

TechnicalEvent

Classified As

Business Event

Classified As

Entity

State Information Model

Event Source

Event Channel

Event Processor

Event Sink

Contains

Defined By

Change May Trigger

Primarily Transfers

BusinessProcess

Primarily Impacts

Published By

Streamed onto

Analyses

Destined For

Produces


Events are an occurrence of behaviour at a specific point in timeA EDA reference architecture classifies events into Technical Events and Business EventsTechnical Events are primarily used to transfer entity state from a source to a target. This is usually triggered by an update to entity state in an enterprise system.Business Events are put into context of a business process. A business event is used to react to the behaviour exhibited by the event. A business event trigger may be based on an update to entity state (simple business event)May be based on detecting patterns in streams of data including the absence of certain conditions (complex event scenario)Entities are enterprise object that are typically represented by an information modelIt is important to align events information and entity information using a consistent modelAn event source is just that, a source of an event stream.An event sink is a destination for an event.An event channel streams events for consumption.An event processor consumes event streams to potentially produce more events

What about Coherence?

Entity

State

Event Sink

Contains

Stores

Is An

Event Channel

Hosts

Event Source

Is An

EventsStores

Coherence


Coherence can play a key role in an event driven architecture byStoring entity state for processingStoring events for availability as an event sinkHosting event channels for distributionBeing a source of events using inbuilt features, which can then be connect to streams for processing

Coherence Features to Consider


Cache Considerations


Data ProcessingOptions and Considerations


The inverse of cachingSends the processing (e.g. EntryProcessors) to where the data is in the gridStandard EntryProcessors provided Out-of-the-boxOnce and only once guaranteesProcessing is automatically fault-tolerantProcessing can be:

Targeted to a specific keyTargeted to a collection of keysTargeted to any object that matches a specific criteria (i.e. Filter)

Features in Context


Accessing PNR Data On Demand

Amadeus

Reservation

Customer

Service Bus

CustomerReservation

On Demand Consumer 1

On Demand ConsumerN

Reservation

Customer

Reservation

Txn Volume = $ Cost


Accessing PNR Data on Demand

Amadeus

Reservation

Customer

Service Bus

CustomerReservation


On Demand ConsumerN

Reservation

Customer

Reservation

Result Cache


Accessing PNR Data on Demand

Amadeus

Reservation

Customer

Service Bus

CustomerReservation


On Demand ConsumerN

Reservation

Customer

Reservation

Read only Entity Cache


Event Driven PNR Data

Amadeus

Reservation

Customer

Service Bus

CustomerReservation


On Demand ConsumerN

Reservation

Customer

Reservation

Entity Cache

Event Consumer 1 Event ConsumerN

Reservation

Customer

Reservation

Event BusCustomerReservation


Caching PNRs in Coherence

Amadeus

Sell (RES)

Fly (CM)



PNR ABC123Created

XML

PNR ABC123Add Flight Leg

XML

PNR ABC123Add Ticket

XML

PNR ABC123

XML



XML?


Coherence POF

The Portable Object Format (advanced serialization)

•Simple Serialization Comparison•In XML

•<date format=“java.util.Date”>2008-07-03</date>•47 characters (possibly 94 bytes depending on encoding)

•In Java (as a raw long)•64 bits = 8 bytes

•In Java (java.util.Date using ObjectOutputStream)•46 bytes

•In ExternalizableLite (as a raw long)•8 bytes

•In POF•4F 58 1F 70 6C = 5 bytes

Slide courtesy of Oracle Product Management



XMLPOJOwith POF

Serialisation


Transforming outside of Cache

XMLPOJOwith POF

Serialisation

J2EE Application Server



MDB Cluster

...

Message Volume

Scale Out Message Processing


Transforming inside of Cache

XML

Coherence Node Coherence Node Coherence Node

Task Processing

...

Message Volume


Processing Task

Processing Task


Approach should consider• Upfront infrastructure costs

– Large volumes and message size impact on initial infrastructure

• Operating stack and associated cost– Message transformation should be low cost– License cost if transformed in cache (albeit not large)– License cost if transformed using MDB potentially a lot larger– Opportunity cost to use cache processing power for more important things ie Analytics– Operational management skillset cost if using open source stack

• Scalability points and capacity– Scaling of data capacity versus processing capacity

• Processing parallelism and tradeoff with guaranteed sequencing


Message Sequencing

PNR ABC123Created

XML


XML


XML

PNR ABC123

POJO

Coherence Coherence Coherence

...

Message Volume


Task Processing

Transform Add Flight

LegTranform

Add TicketTransform

Create


Message Sequencing


...

Message Volume


Task Processing

Transform Add Flight

LegTranform

Add TicketTransform

Create


Message Sequencing


...

Message Volume


Task Processing


POJO


POJO

PNR ABC123Created

POJO


Message Sequencing


...

Message Volume


Task Processing

PNR ABC123Created

POJO


Incubator Processing Pattern

http://coherence-community.github.io/coherence-incubator/11.1.0/processingpattern/index.html




Coherence And Event Processing

Entity Cache

Event BusCustomerReservation


Technical Events vs Business Events

PNR Create

PNR Passenger

Add

PNR Flight Leg Add

PNR SSR Add

PNR Passenger

Contact Add

PNR Ticket Add

Business Class Passenger Requests

Special Service


Bespoke Event Processing using ClientCache Client

Entity Cache Event Listener

Cache Server

Task Execution

Event Listener API

Processing Session

Data Changes

Old New

Create Update

New Old New

Submit Event Determination

Tasks

Delete

Handle Task Outcome

Old

onCreate()onUpdate()onDelete()

Task Factory

Entity Cache

Event Cache

Event Cache Entry

Worker Manager API


Bespoke Event Processing using Server

Cache ClientEntity Cache Event Listener

Cache Server

Task Execution

Event Listener API

Processing Session

Data Changes

Old New

Create Update

New Old New

Submit Event Determination

Tasks

Delete

Handle Task Outcome

Old

onCreate()onUpdate()onDelete()

Task Factory

Entity Cache

Event Cache

Event Cache Entry


Oracle Event Processor


Event Processing using OEP

Events Events Query push-down

OEP Server

model display


Oracle Event Processing Example

OEP Server(s)

Cache Server(s)

Entity Cache

Technical Event Cache

ProcessingTasks

Business Event ProcessorJMS Adapter

Event Bus

Business Event

Channel

Event Consumer

JMSMessage Channel

Entity Processor

Technical Event

Procesor

ProcessingTasks

Technical Event

Channel


Coherence to Store Entities and Events


Coherence to Store Entities and Events <wlevs:caching-system id="CoherenceCache" provider="coherence" advertise="true" /> <wlevs:cache id="PlatformTechnicalEventCache" caching-system="CoherenceCache" value-type="PNRTechnicalEvent" key-properties="sourceMessage"> </wlevs:cache>

<wlevs:cache id="PlatformEntityCache" caching-system="CoherenceCache" value-type="PNR" key-properties="reloc"> <wlevs:cache-listener ref="PNREntityCacheListener" /> </wlevs:cache>

<cache-config xmlns="http://xmlns.oracle.com/coherence/coherence-cache-config" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://xmlns.oracle.com/coherence/coherence-cache-config coherence-cache-config.xsd">

<caching-scheme-mapping>

<cache-mapping> <cache-name>PlatformEntityCache</cache-name> <scheme-name>PnrDistCache</scheme-name> </cache-mapping>

<cache-mapping> <cache-name>PlatformTechnicalEventCache</cache-name> <scheme-name>MsgDistCache</scheme-name> </cache-mapping>


Coherence as an Event Source

<wlevs:cache id="PlatformTechnicalEventCache" caching-system="CoherenceCache" value-type="PNRTechnicalEvent" key-properties="sourceMessage"> <wlevs:cache-listener ref="PNRTechnicalEventCacheListener" /> </wlevs:cache>


Coherence as an Event Sink

<wlevs:event-bean id="PlatformMessageToTechnicalEventBean"class="au.com.intelligentpathways.event.bean.PlatformMessageToPNRTechnicalEventBean"> <wlevs:listener ref="PlatformTechnicalEventCache" /> </wlevs:event-bean>


Coherence for Event Processing

<wlevs:config xmlns:wlevs="http://www.bea.com/ns/wlevs/config/application" xmlns:jdbc="http://www.oracle.com/ns/ocep/config/jdbc"> <processor> <name>PNRBusinessEventProcessor</name> <rules> <query id="BusinessClassPassengerAddsSSR"> <![CDATA[

select 'BUSINESS_CLASS_SSR' as eventType, te.reloc as id from PNRTechnicalEventOutboundChannel [now] as te, PlatformEntityCache as e where te.reloc = e.reloc and te.eventType = 'PNR_SSR_ADDED' and e.bookingClass = 'BUSINESS' ]]> </query>

</rules> </processor> <channel> <name>BusinessEventChannel</name> <selector>BusinessClassPassengerAddsSSR</selector> </channel></wlevs:config>


Take away points

• Event Driven Architecture to remove on-demand access to core system

• A lot to consider on approach• Coherence front and centre technology• Coherence and OEP work together nicely• Both technologies lay the foundation for real

time response to customer and organisation behaviour


Thank you for your time.


coherence in an event driven architecture

Technology

pnr pnr

event streams

event driven pnr data

business event trigger

pnr passenger

event channel streams

event driven architecture

pnr data changes