event driven architectures
TRANSCRIPT
Event Driven Architectures
Avinash Ramineni
• What is Event Driven Architecture• How is EDA related to SOA• Event Capture/Generation• Event Processing• Event Delivery Guarantees• Complex Event Processing• Events, Internet of Things and Big Data
Agenda
• Significant change in the state of an Object– Bank Transaction – Students grade gets posted– New Order gets created– Customer Profile gets updated– Change in Heart Beat
• Characteristics of an Event– Events are broadcast– Communications are asynchronous– Events are fine grained
Event
• Handling of the Events as they occur and the transfer of events among the systems in a loosely coupled way
• Enables Situation Awareness and enables sense-and-respond behavior
Event Driven Architecture (EDA)
• Characteristics of EDA – Producer/Consumers are not aware of each other– Little or no statefullness– Distributed– Loosely Coupled– Platform and Language Independent– Reliable– Fast and Highly Scalable
Event Driven Architecture (EDA)
EDA extension of SOA
Computing with event objects
REST, URIs and HTTP
Event Processing
RPC-style SOA
Event-driven SOA
SOA
Web Architecture
Component software done right
Web-oriented architecture (WOA)
Non-SOA EDA and CEP
Simple Web sites
• EDA similar to SOA facilitates Agility by using Modular Design and Separation of Concerns
• EDA fits well into organizations that have some SOA infrastructure in place
• EDA adds another dimension / Possibilities to SOA– Events are generally stateless and do not have much content
• While processing the events that do not have much data in them, the consuming applications call the services exposed to get the actual data
• Publish – Subscribe Model– Similar to Asynchronous Messaging Architectures
EDA extension to SOA
• Application Generated – Application generates an event when a business transaction
completes– Events generated by parsing the access logs/ server logs– Polling web pages – capturing the events from the services
• Database Generated – Database Trigger generated Events
• On insert/update/delete– During database replication
How to generate Events
• The network is reliable• Latency is zero• Bandwidth is infinite• The network is secure• Topology does not change• Transport cost is zero• Network is homogeneous
Fallacies of Distributed Systems
• Send an email when a customers profile gets updated Customer Profile– Assumptions:
• There is an Application A that provides a updateProfileService• There is an Application B that provides a sendEmailService• All the Events are delivered with Once and Only Once SLA and the Events
are processed
Usecase: 1
A Event Channel
DB
Update DB
Profile Updated Event
B
• The overhead involved in capturing events must be as low as possible to avoid impairing the performance of a business transaction.
• Business Transaction and the Event generation needs to be in the same transaction– 1,2 needs to be part of a same transaction
• Both 1 and 2 should succeed or both should fail
Considerations while generating Events
A Event Channel
DB
1. Update DB
2. Profile Updated Event
B
• Successful Update of Profile -- Call a service to publish a Profile update Event
• Can a database transaction and Web service call be made atomic?
Usecase: 1 – Web service
try{ 1. Update Profile on DB 2. Call web service to publish profile updated event} catch(Exception e){ rollback();}commit;
try{1. Update Profile on DB
} catch(Exception e) { rollback(); }commit(); 2. Call web service to publish profile updated event
• Can Compensating Transaction work ?
• Successful Update of Profile -- Put a message on to Message Broker
Usecase: 1 – JMS
A Event Channel
DB
1. Update DB
2. Profile Updated Event (JMS message)
B
• Possible to make 1 and 2 into a single transaction but it requires – Distributed Transaction.
• Follows 2 Phase Commit protocol• Very Chatty protocol and does a lot of logging to be able to
recover from any failure scenario.• Too much overhead to 99.9% of the cases to handle less than
0.1% of cases• Increases the chances for deadlocks on the database. • Lowers the overall performance of the system.• Bane of scalability. Grinds the entire system to halt by adding
overhead to each and every transaction.
Distributed/XA Transactions (1)
• Availability of the Systems goes down.• XA Configuration is complicated • Difficult to test.• Many People tend to believe that using JTA implementation of
transaction manager will take care of a XA transactions• Many a time people end up using JTA Manager even while
dealing with single resource.
Distributed/XA Transactions (2)
• Successful Update of Profile -- Put a message on to Message Broker
Usecase: 1 – Local Database
A Event Channel
DB
1. Update DB
3. Batch process to send events to Event Channel B
• 1 and 2 are in same transaction. 3 can be retried multiple times till they succeed.
2. Store Event
• Successful Update of Profile -- Put a message on to Message Broker
Usecase: 1 – Local JMS
A Event Channel
DB
1. Update DB
3. Publish to Event Channel(JMS message)
B
• Possible to make 1 and 2 into a single transaction without a Distributed Transaction as long as Queue is backed by a same database
queue
Send Event to Local Queue
2
• Successful Update of Profile -- Put a message on to Message Broker / Call a web service to publish an Event
• Have a Reconciliation Process in place to verify that there is an event generated for every Business Transaction– Have a unique Id (may be stored in DB) along with update and use that
Id as Correlation Id with the Events Archive– If it events don’t tally up, recreate those events
• Use the DB triggers to generate events – Write to a different local table – Call a web service to send the event to event channel (have seen
people do that)
Usecase: 1 – Other Ways
• Event Delivery Guarantees– Reliability• At least Once
– Duplicate events can be delivered
• At most Once– Some events can be lost
• Once and only Once– Each is delivered Exactly once
– Order of Delivery• In Order Guaranteed• In Order not Guaranteed
Event Delivery
• Processing an Event and associated Business Transaction needs to be in the same transaction
• 3,2 needs to be part of a same transaction• Both 3 and 2 should succeed or both should fail
Considerations while Processing Events
Event Channel
3. Acknowledge Event Process success
1. Profile Updated Event
B
2. Send Email
• On Profile Update Event -- Call a service to send Email and on success acknowledge process successful
Usecase: 1 – Consumer (1)
onMessagetry { sendEmail(); jms.commit()}catch (Exception e) { jms.rollback()}
• What if jms.commit() fails ??
• If Jms.commit() fails , message gets delivered again
Usecase: 1 – Consumer (2)
onMessagetry { if I have not processed this message successfully before { do some stuff in the database / Idempotency Logic /JMSRelivered flag jdbc.commit; } jms.commit()}catch (Exception e) { jms.rollback()}
• Idempotent Operation– An idempotent operations means that the result of a successful
performed request is independent of the number of times it is executed.
• Design consumer to be an Idempotent Consumer– By Understanding the Business rules it can be achieved relatively easy
• Good Practice to make all the write operations Idempotent !!• Idempotency logic can be built on Event Ids, Business Ids –
orderId, customerId, etc based on the Business scenario
Work Around for At-Least-Once
• In-order delivery of events– Limit the number of event publisher/consumer instance
for these specific events to 1 • Brings up questions like if one of the event is having an issue – all
the events of that event type can come to a stand still
– AVOID the need for In-order Delivery of Events• By limiting the content in the Event , we can avoid a lot of cases
where In-order delivery is required• Using Message Selectors to select and aggregate the events can
solve a few of the cases
What If In-order Delivery is required
• Student Information System (SIS) maintain a list of students in a course roster. The students can be added or dropped from the course roster. SIS sends an event to Learning Management System (LMS) when the roster gets updated.
• Lets us assume that SIS is sending the course roster as part of the event.
• Processing of this Event requires the Roster Change events to be delivered in order to maintain the data from getting corrupted.
Usecase: 2
Usecase: 2 – Requires In-Order
SIS Event Channel
DB
2. Deliver Roster Change Event
LMS
LMS DB
1. Roster Change Event
• Remove the roster data from the event and just pass a roster Id
• On Delivery of the Event , the LMS system would use the rosterId from the event, makes a service call to the source system (SIS) to get the latest Roster data and builds the classes in LMS
• Even if the events are delivered out of order , the roster data on LMS will be always be the latest.
• By making the processor on LMS idempotent, we can even avoid the database update
Usecase: 2 - Solution
Usecase: 2
SIS Event Channel
DB
2. Deliver Roster Change Event
LMS
LMS DB
1. Roster Change Event
3. Get the latest roster for the Id
4. Update the LMS database
• Make Event Publishers responsible for making sure that events are generated for every Business transaction and are published to the event channel
• Make Event Consumers responsible for making sure that they react to the Stimuli from the event
• Dashboards in place to make sure failed Events are detected and acted upon
• Collects Stats to see if the events need to be made finer or coarser
• Maintain Event Catalogs and self service capabilities to subscribe to events
Best Practices
Complex Event Processing (CEP)
Traditional BI finds "needles in haystacks."
Event-driven CEP finds "needles" in continuously arriving streams of "hay."
• CEP tap the events happening with in an organization and provide a situation awareness and enable better and faster decisions– Identify complicated events that are inferred from event pattern
detection, event pattern interpretors and so on– Ex: Monitoring – capture events from ticket system, network load,
performance and send out notification bring up new servers into the pool
• With Big Data hype and with the “capture-it-all“ approach and IoT , EDA and Event Driven programming will get a huge boost in the coming years.
• Events and EDA due to their characteristics will be very well suited for Ubiquitous Computing
• We might start seeing an advent of Complex Event Applications by being used by various gadgets – The concepts of EDA and Events will be a perfect Match for gadgets like Basis ,FIT
Events ,Big Data and Internet of Things
