SOA è MSA

Event Driven Application Extensions by Design

Todd W. FritscheCerner CorporationVP: Components and Platforms

• Headquarters: North Kansas City, MO, USA

• 2017 revenue: $5.1 billion

• Cumulative R&D investment of over $6.3 billion

• More than 27,500 Cerner associates based in 26 countries worldwide

• Cerner solutions are contracted at more than 27,000 provider facilities in over 35 countries

• “Cerner” comes from the Latin word meaning “to discern”

My Career

Student Programmer Architect Entrepreneur CTO Executive

SOA vs MSA

Source: https://dzone.com/articles/microservices-vs-soa-is-there-any-difference-at-al

SOA

• What We Got Right (mostly)• Simplification / Composition• Governance / Repository

• What We Got Wrong• EDA: Event Driven Architecture• XA Transactions / ACID / 2-phase commit• Deployment

MSA

• What We Got Right• Simplification / Composition• Container Deployment• Compensating Transactions

• What We Are Getting Wrong• EDA: Event Driven Architecture• Governance / Dictionary

• Saving this for another day!

Open Systems è Open Platform

• Techniques for “opening” your application• SQL Report Writers

• EDI: Electronic Data Interchange• HL7, X12

• API: Application Programming Interface• API Gateways• FHIR

• EDA: Event Driven Architectures• PUB / SUB• Application extension via Complex Event Processing + Orchestration + Integration

Event Driven Application Extension

Is this an event I’m interested

in?

What actions

do I need to take?

Execute action

Execute action

Execute action

Why Open Your App to Extension?• Federation of domain knowledge•Why do we like open source?• Your clients / users know their business better than you

do

• Cost• New feature / function via DEV teams is expensive• And slow!

• Stickiness• Clients / users become “invested” in you app / platform

HEP Business Challenge• Be more open than our competition

• Enable our clients to integrate with non-Cerner solutions• Typical 200 bed hospital has >50 back office applications

• Enable our clients to adapt to their unique environment• Community Hospital, Academic Medical Center, Public Health Systems

(NYCHHC), Non-Profit, Out Patient Facility, Physician Group• Unique payer mix• Complex regulatory environment

HEP Block-itecture

hawkpod

Virtual - LinuxHawk

Hawk

BW Node

Citrix Farm

Virtual - LinuxCitrix

GEMS Log Viewer

BE Studio

BW Studio

prodpod

Virtual - LinuxMsg Logger

Msg Logger

Virtual - LinuxElasticSearch / Kibana

Elastic Search

Kibana

hawklog stash hawklog stash

Admin Server

hawklog stash

TEA

MySql

Virtual - LinuxBE / BW Runtime

BE Inference

BE Cache

BW Node

hawklog stash

BW Agent

testpod

Msg Logger

Msg Logger

ElasticSearch / Kibana

Elastic Search

Kibana

hawklog stash hawklog stash

BE / BW Runtime

BE Inference

BE Cache

BW Node

hawklog stash

BW Agent

Legend:

Virtual Servers

Physical Servers

hawk log stash

EMS

Multi Tenant

Single Tenant

TEA UI

Log Admin

Virtual - LinuxEMS

EMS

hawklog stash

Virtual - LinuxEMS

EMS

hawklog stash

Source Control

Web Server

Supporting Infrastructure Components

Active Directory

Netscaler

VMWare

NFS Storage

DNS

Chef Server

Alert Logger

CA Unicenter

SMTP Server

Certificate Server

Artifactory Server

Prod EMS Message Volume (Inbound to Queues)

Month POD Number clients Live Clients Min Msgs per day Max Msgs per day Avg Msgs per day Max Msgs per second

Nov 2017 Total 60 36 32,668,662 110,276,462 75,398,998 1,276.35

Dec 2017 Total 60 36 34,199,028 118,886,188 76,577,124 1,376.00

Jan 2018 Total 61 45 38,368,468 115,423,027 80,520,078 1,335.91

Feb 2018 Total 61 49 42,632,039 121,729,234 88,894,128 1,408.90

Mar 2018 Total 63 53 42,912,978 121,924,341 92,710,254 1,411.16

Apr 2018 Total 65 53 41,420,924 118,326,558 87,313,854 1,369.52

May 2018 Total 65 53 43,647,721 131,767,909 94,409,195 1,525.09

June 2018 prodpod001 15 10 10,555,538 33,480,578 24,323,563 387.51

June 2018 prodpod002 12 9 9,325,651 29,851,506 20,655,960 345.50

June 2018 prodpod003 12 12 6,576,923 18,522,185 14,166,277 214.38

June 2018 prodpod004 16 15 9,783,023 25,926,490 20,660,376 300.08

June 2018 prodpod005 1 1 3,966,648 12,999,955 10,021,676 150.46

June 2018 prodpod006 10 6 3,945,919 12,252,297 8,435,044 141.81

June 2018 Total 66 53 44,153,702 133,033,011 98,262,895 1,539.73

ELK Stack• Logs

• Logstash installed on each server – gathers log files and sends to ElasticSearch clusters

• Allows log messages from all servers to be consolidated and search in one location –no logging into each server to view logs

• EMS messages• Java app that consumes sys.monitor EMS topics and sends data to ElasticSearch• Allows full tracking of messages through EMS (receive, send, acknowledge)

• Audit messages• “Special” log messages that are produced by BE and BW that audit the execution of

workflows• Created BW palate to produce the audit messages

• Kibana• Tools to search and visualize the data

PRODPOD001 SmartFlows Statistics (last 7 days)

Hawk Monitoring• 90+ rule base alerts defined• Process alerts – is process running?• Log watchers – Warnings, Errors, Fatal messages in logs• Threshold alerts – Low, Medium, High alerts

• CPU, Memory Disk• EMS pending messages

• Hawk agent installed on each server• Sends alerts to Hawk server via EMS messages

• Hawk Web Console – used to view active alerts• Alerts are sent to internal ticketing system• Created BW app to read event.dat file

HEP Model Content:Revenue Cycle SmartFlows

What are Revenue Cycle SmartFlows?• SmartFlows provide a workflow framework approach to automate

Business Office follow up activities, based on client-configurable rules, leveraging HEP.

• Framework Components:• Event Publication

• Implemented as application published events or events raised via database triggers, whenever the application makes updates

• Standardized Data Model • A consistent data model available for all published events that can be used across rules

• User-friendly decision table rule implementations• Excel-like decisions tables that evaluate the Standardized Data Model and determine what

action(s) if any should be taken• Application web services

• Delivered fully configured and ready to execute based on decision table outcomes

SmartFlows Implementation • Revenue Cycle Smart Flows are delivered off by default for all clients.• Enable Smart Flows and specific Decision Tables via BE Global

Variables.

SmartFlows

• Add Receivable Group Comment• Save Guarantor Collection Profile• Save Receivable Collection Policy for

Receivable• Save Receivable Collection Policy for

Receivable Group• Save Receivable Collection Policy for

Guarantor Account

• Save Receivable Dunning Level Code• Transfer Payer Balance• Add Collection Hold • Remove Collection Hold • Insurance Management• Payer Adjustment• Charge Management

Revenue Cycle SmartFlows Process

HEP

(RCS

F) R

untim

e Co

deSF

HEP

(RCS

F) M

emor

y Ca

che

Full Process

Event

Received by BE

Event Preprocessor

Enrichment Preprocessor

Service Specific Rule

RCSF Event

DB Concept

Instances

Descision Table

BW Service Wrappers

Some action in SFSoarian

Financials DB

Events and DB Adapter Events

Event Processing “Trade Offs”• One Event or Many Events?• A single RULE to process ALL events (brute force method)• Different RULES for EACH event (take advantage of Rete)• DECIDING FACTOR FOR US: The effect of RULE deployment on availability

• Stateful or Stateless Event Processing?• Multiple copies of data == potential for “stale” data• Inflating data model for each event == extra processing

Revenue Cycle SmartFlows Event• Aka the “Super Event”• Created from the inbound event (application or db adaptor)• Contains:• System fields needed to define the event

• E.g., eventName• Event fields from each type of inbound event

• E.g., the encounterIdentifier field from Encounter events

• Needed for several reasons:• We do not know which event will be needed for the use-case• We need to have a consistent event for the rules to run from

Database Adaptor Events• Allows us to create new events for use with HEP

• Allows to create ability for third-party apps to generate events

• Utilize TIBCO Database Adaptor functionality

• Events are generated based on database changes

• Once event hits BE – treated like any other event

DB Adaptor Events

BusinessEvents Concepts• In-memory representations of a set of

data• Think of: database table definition• Contains:• A set of fields (aka, “properties”)• Their associated data types

• Example: Appointment Concept• Patient information• Appointment start date and time• Appointment location• Providers• etc…

RCSF DB Concept Enrichment• Done in Application Preprocessors

• Preprocessors:• Rule functions• Fire before all other rules, rule functions

• Two steps (preprocessors):• First:

• Pulls event fields off of event, • Creates “SuperEvent”

• Second: • Does all SQL queries to create instances of database concepts• Asserts all database concept instances• Asserts SuperEvent

Data Evaluation Using Decision Tables• Enriched data concepts can be

used as conditions in Decision Tables (DT)

• Based on evaluation of conditions, appropriate calls to Public Services are invoked

• One or more actions invoked per event, based on which DT actions evaluate as true

Deciding What to Do in BE Rules

• BE rule evaluation can follow these patterns:1. One rule for each event type condition– WHEN field =

value2. One rule that handles all conditions with a very long

IF/THEN statement.• IF comment code = 1234 THEN hold reason code = 9876• ELSE IF comment code = 3456 THEN hold reason code =

87653. Fire a single rule for all events and evaluate conditions via

a decision table – IF/THEN in table form

Decision Tables

• Visual representation of a rule• Each Decision Table performs a specific action• Contains a set of conditions and a set of actions• If all of the conditions are true• Then perform the action

• Conditions can be:• Defined fields from the DB concepts or super event• Model HEP rule functions

• RevenueCycleSmartFlows.Functions.BlPerPtAlertCdHasAll(BlPerDetail,“123,233")• Custom functions

Decision Tables

Service Wrappers

Service Wrappers• Makes service calls to Cerner applications easier• Wrappers exist in BE as Rule Functions and in BW as a Sub

Process• Service Queues: • com.cerner.entp.{hhrr}.{env}.private.bw.sf.servicewrapper.<service

name>_v<#>_q

Configuring Service Calls Without a Wrapper• Import service WSDL into BW• Create a reference to the service• Create the SOAP reference binding• Configure the SOAP reference binding• Add SMS Namespace• Add soapAction• Invoke the service call• Bind the evoke step to the reference binding• Configure service inputs and outputs• Configure error handling• Configure auditing and logging

Configuration with a Service Wrapper• Import service WSDL into BW• Create a reference to the service• Create the SOAP reference binding• Configure the SOAP reference binding• Add SMS Namespace• Add soapAction• Invoke the service call• Bind the evoke step to the reference binding• Value service inputs and outputs• Configure error handling• Configure auditing and logging

Available Revenue Cycle Services

• 55 Services Total• 14 Guarantor Billing Collection• 3 Insurance Collection• 21 Receivables Management• 5 Worklist Item Management • 2 Collection Hold for Receivable Group• 10 Encounter

Wrap Up• Lessons learned• EMS is awesome when you pay attention to IOPS • Chef recipes and TIBCO component installs + upgrades are hard• BW DB Adapter sounds scary but works / performs well• ZDT upgrades are REALLY hard

• Achievements• ZDT upgrades• Happy clients

• What’s next• Containers + elastic auto scale out• Continued development of SmartFlows

• Q&A