1

Define Enterprise Integration strategy

- Author

Bhawani Nandan Prasad

(bhawani_Nandan@yahoo.com)

- Program\Engagement Management,

Enterprise Architect, Information architect, Data

Architect, BI Architect and Application Architect

• Master of Business Administration• Diploma in Senior Management• Bachelor of Engineering in Computer Science

2

Integration strategy statement

Integration of systems and applications is core to the success of technical interoperability. Encore’s integration strategy is that developers use tools and technologies that are “fit for purpose” i.e. are applicable for the task at hand and are documented within the Encore EA standards. All solutions must use approved technologies with established, repeatable patterns that enable the Encore technical team to respond in a timely and well understood fashion.

3

EI Methodology

• Integration of systems and applications is core to the success of technical interoperability. Our Approach – Define the current state for the tier 1 application. Identify the existing integration points, functional or non-functional. Identify the current integration characteristic, is it event driven, real time and synchronous, or is it delayed and in bulk mode. What could be the functional state, real time, event driven, bulk or one record at a time, back end data synchronization, exposed reusable services, etc.

• Deliverables: Current integration points, for each application potential integration points or extensions.

• Steps/Plan

• Define the existing functional/non-functional integration points

• Define the current technical stack for the integration

• Define the end state for functional integration.

4

EI Macro / Micro Level Design

Integration

Style/ Criteria

Batch File

TransferETL

Direct Data

Access

Point-to-

Point

Messaging

(Event Driven

Architecture /

Pub-Sub)Services/ ESB

Transport

Protocols

• FTP

• SFTP

• FTPS

ProprietaryODBC/JDBC/

Proprietary

JMS /

Websphere MQ

JMS /

Websphere MQHTTP/ HTTPS /FTP

Business Logic NoneTransforms/

ValidatesDuplicated Distributed Distributed

Distributed and

Composable

Data Format

• Industry

• Vendor

• Own BU

• Industry

• Vendor

• Encore

• Stored Procedure

• DAL Pattern

• ORM Framework

• Industry

• Vendor

• Encore

• Industry

• Vendor

• Encore

Industry Standards

• SOAP

• XML

• JSON

Coupling Loose Loose Very Tight Loose Very loose Loose

Topology 1 to 1 Many to Many 1 to 1 Many to Many Many to Many 1 to Many

Vendor /

Product Lock inLow High Medium Medium to High Medium to High Medium to High

Language /

Platform

Dependency

Low High Medium Medium to High Medium to High Low

5

Integration Method Comparison

Integration

Style/ Criteria

Batch File

TransferETL

Direct

Data

Access

Point-to-

Point

Messaging

EDA Services/ ESB

Availability Medium High Medium High High Medium

Flexibility Low Medium Very Low Medium Very High Very High

Configurable QOS No Yes No Yes Yes Yes

Payload Size High High Medium Low Low Medium

Frequency Low Low High High Medium Medium to High

Maturity of Tech

and Tools (1-5)5 5 5 5 2 3

TCO (maintenance,

risk)High

High set up

Maintenance

lower

High Medium Medium Low to Medium

Complexity Low MediumLow to

Medium

Medium To

HighHigh Medium to High

Assured Delivery/

ReliabilityLow High High

Medium To

HighMedium To High Medium to High

6

Integration Method Comparison (cont.)

Integration

Style/

Criteria

Batch File

TransferETL

Direct Data

Access

Point-to-Point

MessagingEDA Services/ ESB

Capabilities • Datagram

• Datagram

• Fan-In

• Fan-Out

• Queue based

• Data Integration

(ETL) Solution

• Datagram

• Fan-In

• Fan-Out

• JMS

Publish/Subscribe

• Topic based

• Subscriber/Recipi

ent List

• Request/Reply

• Queue based

• Datagram

• Fan-In

• Fan-Out

• JMS

Publish/Subscribe

• Topic based

• Subscriber/Recipie

nt List

• Request/Reply

• Queue based

• Datagram

• Fan-In

• Fan-Out

• JMS Publish/Subscribe

• Topic based

• Subscriber/Recipient

List

• Request/Reply

• Queue based

Tools

Generic

• JScape

• MoveIt

Platform Specific:

• CD Connect

• Transit

• ETL

Decomissioned:

• FileZilla

• CoreLite

• SSH

• Shared Drives

Generic:

• DataStage

• CastIron

Platform Specific:

• DataLoader

(Salesforce)

• SSIS (MSSQL)

• Change Data

Capture

(MSSQL)

Java:

• Websphere

Microsoft:

• IIS

• LinkServer

(MSSQL)

• Websphere MQ • Websphere MQ

ESB:

• Websphere MQ

• CastIron

• DataPower

Services:

• Websphere WS (Java)

• WCF/ IIS (Microsoft)

7

Integration Method Comparison

Integration Styles

8

Appendix A

File extract produced by Application A extract program. Transferred physically (media) or via FTP. File read by Application B load program

Custom code usually written in each application to produce and process extract files.

9

App A App B

Batch File Transfers (FTP)

ETL Software provides set of tools and infrastructure for extracting data from source system databases or files, apply transformations and loads resulting data into target system.

Often the extracts are “intrusive” database reads, but may also use extract files generated by the Application

10

App A App B

ETL

Extract, Transform, Load (ETL)

Direct access/update by Application A to Application B data

• Usually uses ODBC/JDBC

• Data “owned” by one of the applications

Shared (Master) Databases• Multiple applications

• Update a shared database

11

Application

B Data

Application BApplication A

SQL / ODBC /

JDBC etc

Shared

Data

Application BApplication A

SQL/ODBC/

JDBC etc

SQL / ODBC /

JDBC etc

Local Admin

Direct Data Access

App A App B

RMI / IIOP

12

Language specific API exposed by Application A called synchronously and directly by

other applications, e.g. RMI

Calls are usually “fine-grained” method level, e.g. retrieving a single data item

Remote Procedure Call (RPC)

App A App B

Queue Management Software

13

Point to Point Messaging Integration– Asynchronous message queuing between applications

– Provides message reliability / guaranteed delivery

– Both endpoints have embedded queuing software

Point to Point Messaging

Application A Application B

Application C Application D

Integration

Broker

14

Messaging Hub or hubs with routing and transformation

capability. Usually uses underlying message queuing

software for transport. Often includes pre-built ‘adapters’to vendor business applications

EAI (Integration Brokers)

Application A

Application B

Application C

Application D

Pub-Sub

Manager

Topic A

Topic B

Topic C

15

“Event” published by an application on a channel or “topic” to a “subscription manager” which distributes the event information to all of the current open subscribers. Publishers and subscribers often are not aware of each others identity. Often uses underlying queue management software for transport

EDA (Event Driven Architecture / Pub-Sub)

Application A

Application B

Application C

Application D

Look up

ESB

Services

Registry

Look up

Orchestration

Engine

SecurityWeb

Services

Mgmt

16

Service interface exposed by business applications available for use/reuse over standard protocols.

ESBs provide routing, transformation and other services. Services Registry provides service look up and location transparency. Standardized security, systems management etc. Orchestration

SOA (Services / ESBs)

Integration Pattern Topics

Review Integration Patterns

• Datagram

• Fan-In

• Fan-Out

• JMS Publish/Subscribe– Topic based– Subscriber/Recipient List

• Request/Reply– Queue based– RPC

• Data Integration (ETL) Solution

17

Datagram Integration Pattern

Requester SourceIntegration

Layer

18

Considerations

• When send and forget (asynchronous) integration is sufficient

• Simple pass-thru integration

• Loosely coupled end points

• Limited reusability

• Error handling procedure

• Data retention

Fan-In Integration Pattern

Integration

Layer

Source

System

Target

System

Source

System

Source

System

19

Considerations

• Multiple source systems

• Interoperability between heterogeneous systems

• Loosely coupled end points

• Increases reusability

• Central control and management for mediation and transport

• Supports content based routing

• Latency differs from each source systems

Fan-Out Integration Pattern

Integration

Layer

Source

System

Target

System

Target

System

Target

System

20

Considerations

• Multiple target systems

• Interoperability between heterogeneous systems

• Loosely coupled end points

• Increases reusability

• Central control and management for mediation and transport

• Supports content based routing

Publish/Subscribe – Topic based Pattern

Integration Layer

Publisher Subscriber

Subscriber

Subscriber

TopicPublish Subscribe

Subscribe

Subscribe

Acknowledge

Acknowledge

Acknowledge

21

Considerations

• The most appropriate distributed architecture

• Interoperability between heterogeneous systems

• Increases scalability

• Consumption of data flexibility

Publish/Subscribe – Subscriber List Pattern

Integration Layer

PublisherRecipient

ListPublish

Subscriber

Subscriber

Subscribe

22

Considerations

• The most appropriate distributed architecture

• Interoperability between heterogeneous systems

• Requires routing to a dynamically specified recipients

• Consumption of data flexibility

Request / Reply (Queue) Pattern

Integration

LayerRequester Provider

RequestRequest

Response Response

23

Considerations

• When applications require two-way (synchronous) conversations

• Increases maintenance and support (requires 2 queues per end point)

• Change end points without impacting existing services

Request / Reply (RPC) Pattern

Integration

LayerRequester

RequestRequest

Response Response

Provider

24

Considerations

• Interoperability between heterogeneous systems

• When synchronous integration is required

• Change end points without impacting existing services

Data Integration (ETL) Solution

Extract Transform Cleanse LoadLegacy

Systems

Packaged

Application

Other

Internal

Application

ETL Solution

Transient Data

Source

Manhattan

WMS

DataMart

25

Considerations

• When large data integration is required

• Integration occurs at the database and data source level

• A proven industry standard for data integration

• Flexibility of input/output data formats

Conceptual Model View

26

Client Access Tier

Presentation Tier

Application Service Tier

Integration Tier

Data TierPla

tfo

rm L

ayer

Web Browser Client Server Other

Web Server Terminal Server Mail Server Voice Response

Data Warehouse

Application APIDatabase

Middleware

Message Oriented

MiddlewareETL SOAP

Application Server Package Report ServerPortal Server

HR, Finance,

Planning, Logistics,

Etc.LDAP

The diagram represents the various

tiers of service based on the N-tier

application.

Client Access Tier: Represents the

type of end-user access.

Presentation Tier: Represents the

client touch points and channels of

services.

Application Service Tier: Represents

that application and business logic to

support the functions and services.

Integration Tier: Represents the touch

points between applications and

systems.

Data Tier: Represents the storage of

information that are needed by

applications.

High-Level Application Integration Model

Application ABC

Custom Code Based

Batch

Application XYZ

EAI HUB (SAP XI)

ETL (Ascential)

Fast Asynchronous

Messaging

ERP

Adapter AdapterModule

Module

Module

Module

Adapter AdapterShort Batch Messaging

Queue Queue

Queue

Ap

plic

ati

on

Lo

gic

ETL Based Batch

Staging

Simple DBMS to DBMS Replication

ERP

API

1

23

4

1. Pull data from source system

2. Stage data in a persistent store

3. Process staged data and transform

4. Load data into target system

Point to Point (API) Integration

FILEFILE

FTP

Application Server

Synchronous SOAP

UDDI

Application Logic

27

Legend

Fast Async

Short Batch

Custom Batch

ETL Batch

DB Sync

P2P

FTP

Near Real-Time Interface - CORE

Message based Store & Forward Short Batch - CORE

Custom Developed Batch - CORE

ETL Based Store & Forward Batch - CORE

DBMS to DBMS Replication - SPECIALIZED

Synchronous Point to Point - DECLINING

SOAP

File Transfer Service - CORE

Synchronous SOAP - CORE