© 2014 IBM Corporation

Enable Business Continuity and High Availability

through Active-Active Technology

David Huang, CTO of China Everbright Bank Credit Card

Qian Li Jin, jinql@cn.ibm.com, Senior manager of WebSphere Messaging

Please Note

IBM’s statements regarding its plans, directions, and intent are subject to change

or withdrawal without notice at IBM’s sole discretion.

Information regarding potential future products is intended to outline our general

product direction and it should not be relied on in making a purchasing decision.

The information mentioned regarding potential future products is not a

commitment, promise, or legal obligation to deliver any material, code or

functionality. Information about potential future products may not be incorporated

into any contract. The development, release, and timing of any future features or

functionality described for our products remains at our sole discretion.

Performance is based on measurements and projections using standard IBM

benchmarks in a controlled environment. The actual throughput or performance

that any user will experience will vary depending upon many factors, including

considerations such as the amount of multiprogramming in the user’s job stream,

the I/O configuration, the storage configuration, and the workload processed.

Therefore, no assurance can be given that an individual user will achieve results

similar to those stated here.

1

Agenda

Active/Active Solution Overview

• Business Continuity and High Availability Requirement

• Active-Active Solution and Business Value

• Active-Active Common Architecture

WebSphere Messaging Technology in Active/Active Solution

• Messaging Key Functions

• Flexible Application Design

• High Performance with Persistence

China Ever Bright Bank (CEB) Active/Active Solution

• CEB Credit Card System Background

• Active/Active Architecture

• Active/Active Operation and Site Switch

• Best Practice

2

High Availability in Enterprise

Enterprise Business Requires High Availability

Standby Active-Active

1. Disaster

Recovery

• Restore the

business after

a disaster

2. High-Availability

• Meet Service

Availability objectives

e.g., 99.9% availability

or no more than 8

hours of down-time a

year for maintenance

and failures

3. Continuous

Availability

• No downtime ever

(planned or not)

3

Technologies for Business Continuity

Requirements: Disaster Recovery High Availability Continuous Availability

Replicated

Objects:

Technologies: •Disk Copy •DB Recovery •Transaction Replay

Requirements and trade-offs to consider in selecting technologies:

• What needs to be recovered?: Application Data vs. DBMS vs. Entire Systems

• How long does it take? Recovery Time Objective (RTO): One hours or more vs. few seconds

• How much data could you lose? Recovery Point Objective (RPO): No data loss vs. seconds of data

• Distance required between sites?: 10 meters vs. 100 kilometers

• Hardware Utilization: Standby vs. Active

• Impact on applications: Direct overhead (synchronous technologies) vs. No impact (asynchronous technologies)

• CPU Overhead: Negligible (hardware e.g., PPRC) vs. Proportional to the workload (transaction replay technology)

4

Active/Active Sites Concept

Two or more sites, separated by unlimited distances, running the same

applications and having the same data to provide cross-site workload

balancing and Continuous Availability / Disaster Recovery

Customer data at geographically dispersed sites kept in sync via

synchronization

GDPS/PPRC GDPS/XRC or GDPS/GM Active/Active

Failover model Failover model Near CA model

Recovery time = 2 minutes Recovery time < 1 hour Recovery time < 1 minute

Distance < 20 KM Unlimited distance Unlimited distance

CD1SOURCECD1TABLE

CD1SOURCECD1TABLE

CD1SOURCECD1TABLE

CD1SOURCECD1TABLE

CD1SOURCECD1SOURCECD1TABLECD1CD1TABLE

CD1SOURCECD1TABLE

CD1SOURCECD1TABLE

CD1SOURCECD1SOURCECD1TABLECD1CD1TABLE

5

Active-Active Business Objective

Reduce planned/unplanned outage for the critical applications

Both site A and B run workload

Achieve extremely short data recovery time (mins) and zero

data loss during disaster recovery.

Achieve Continuous Availability of the key applications which

support the critical business services through flexible switching

over and back between site A & B

“Near real time business analysis”

Efficient utilization of IT resources of Site B

6

Active-Active Common Model based on Messaging

Business App

BusinessData

SyncApp

Messaging

SyncApp

Messaging

Business App

BusinessData

Workload Distributor

Sites at a distance

•Cross Site Workload Distribution

•Data synchronization

•Reply on high performance, reliable messaging transmission

•Flexible application design

•Automation & Management

7

Agenda

Active/Active Solution Overview

• Business Continuity and High Availability Requirement

• Active-Active Solution and Business Value

• Active-Active Common Architecture

WebSphere Messaging Technology in Active/Active Solution

• Messaging Key Functions

• Flexible Application Design

• High Performance with Persistence

China Ever Bright Bank (CEB) Active/Active Solution

• CEB Credit Card System Background

• Active/Active Architecture

• Active/Active Operation and Site Switch

• Best Practice

8

Messaging Technology in Active-Active

Data Synchronization is the key component in Active-Active

• Capture transaction change in real-time

• Publish the change in high performance with low latency

Messaging based implementation is proven to be the simplest

way among kinds of methods of data transmission

A high performance, reliable messaging product is needed for

the following requirements:

• Simplifies application development

• Ease of use

• Assured message delivery

• High Performance and Scalability

• Easy of Management

9

WebSphere MQ High Light for Active-Active

Reliability

• Assured message delivery

• Performance

Ubiquitous

• Breadth of support for platforms,

programming languages and API

Loose-coupling

• Location transparency

• Time independence

• Data transparency (with WebSphere

Message Broker)

• Platform independence

Scalability

• Incremental growth

Rapid development

• Reduce Complexity

• Ease of use

Q Manager Q Manager

Message

Queue

Application ZApplication A

Channels

MQ Version Release

– V5.3 Distributed: EOS

– V6 GA: 2005/06

• (EOS 2012/09)

– V7.0.0 GA: 2008/06

– V7.0.1 GA: 2009/09

– V7.1.0 GA: 2011/11 (include MQ for z/OS)

– V7.5 GA: 2012/06 (distributed only)

10

Queue Manager

Synchronization Application Design - Persistence

All messages are important in banking system. Money is included.

Persistent or non-persistent message need to be justified

• Persistent messages will be recovered, non-persistent not

• Persistent messages reduce the cost of developing the application

• Persistent messages increase the cost of processing, need logging

Only use a persistent message if require

MQPUTPersistent Message

MQPUTNon-persistent Message

Application Program

Logging

11

Synchronization Application Design - Syncpoint

Transactions are vital to provide reliable messaging.

Do you need it?• Set of work needs to either all be performed, or all not performed

The Transactional behaviour should be considered in synchronization

application

• The change in source site been published to MQ once and only once

• The message in target site been processed and removed from MQ

May use some other ways to keep data consistency

Actually cheaper to process in syncpoint for persistent messages

MQPUT within

syncpoint

MQPUT within

syncpoint

MQPUT within

syncpoint

MQCMIT

1 2 3 Messages are only

now available

12

Synchronization Application Design - Message Size

The size of messages can be a contentious issue

WebSphere MQ supports a large variety of message size

Queue managers are optimised for 4KB - 100KB message size

• Very large messages are inefficient

• Very small message are inefficient

– The queue manager has to perform the same work for a 1-byte

message as for a 4KB message

Application could leverage this by

• Packaging small transactions into big message

• Split big transaction into small messages

13

Performance Tuning Considerations

Synchronize only the changed data, thus reduce the data

volume

Introduce more parallelism

• Multiple synchronization channels for different type of workload

• More threads in sync application for parallel processing

• Multiple MQ channels to leverage single channel busy problem

Invest to use MQ new feature

• Bigger buffer pools above the bar

• Sequential pre-fetch

• Page set read/write performance enhancement

• Channel performance improvement

14

MQ Buffer pools read ahead enhancement

Symptom: When the number of messages overruns the buffer

pool allocated for the queue, messages are spilled to disk and

must then be retrieved from disk.

The read ahead enhancement enables message pre-fetch from

disk storage and improves MQGET performance.

Available in PM PM63802/UK79853 in 2012 and PM81785/

UK91439 in 2013.

Internal testing shows ~50% improvement with read ahead

enabled (msglen=6KB).

Enable this feature if MQ buffer pool may overrun.

15

Agenda

Active/Active Solution Overview

• Business Continuity and High Availability Requirement

• Active-Active Solution and Business Value

• Active-Active Common Architecture

WebSphere Messaging Technology in Active/Active Solution

• Messaging Key Functions

• Flexible Application Design

• High Performance with Persistence

China Ever Bright Bank (CEB) Active/Active Solution

• CEB Credit Card System Background

• Active/Active Architecture

• Active/Active Operation and Site Switch

• Best Practice

16

Profile of China Everbright Bank (CEB)

Established in August 1992

Registered capital RMB28.21689 billion

By the end of 2008

• Total assets reached RMB843 billion

• Total loans of RMB468.9 billion.

• The operating profit and net profit are RMB7.96 billion and

RMB7.36 billion respectively.

Now

• Covering 45 major cities of 23 provinces, municipalities and

autonomous regions

• 30 branches and 426 banking outlets and one representative

office in HK

• The Bank now has about 17, 000 employees

17

Active-Active Adaptability in Small/Medium-sized Banks

China banks have setup storage based DR solution, but the business

recovery time is too long

Sysplex solution is too expensive, and input-output ratio is not high

Need to consider application based solution, and mix with the storage

based solution

Active-Active is the target model of modern data center

Active-Active is on its way, need a lot of innovation and there’s no copy

model

Not only for mainframe, but heterogeneous and periphery distributed

platform also need to be active-active

18

Background of CEB Credit Card System

CEB credit card system on mainframe gone live in Oct 2011.

This system is based on the VisionPlus (V+) solution by First

Data

(Secondary) V+ MainframeBatch Processing

(Main) V+ Mainframe

Batch Processing

DRNET

Headquarter

Gateway

Finance

Processing in BJ

Finance

Processing in SH

OLTP Processing

OLTP Processing

File Transfer

VISA/MC/JCB .

Non-Finance

Processing

OLTP Batch Terminal Anti-fraud Reporting

Global Mirror

files

Debt-collection

19

Business Requirement of Active-Active

Improve the capacity and availability of the whole credit

card system.

More comprehensive and more efficient services by

payment systems of the banks.

More flexibility accesses, more comprehensive functions

of liquidity risk management, extension of the scope of

system monitoring

Refinement of backup infrastructure

20

2. File Transfer

The target Active-Active System Structure

Both the main system and the secondary system are active

Real data synchronization for OLTP transactions

The main system and the secondary system backup each other

Workload can be taken over in case of planned or unplanned failure

21

(Secondary) V+ MainframeBatch Processing

(Main) V+ Mainframe

Batch Processing

DRNET

Headquarter

Gateway

Finance

Processing in BJ

Finance

Processing in SH

OLTP Processing

OLTP Processing

File Transfer

VISA/MC/JCB .

Non-Finance

Processing

OLTP Batch Terminal Anti-fraud Reporting

3. Global Mirror

files

Debt-collection

Workload Split by

Card BIN, and send to

BJ and SH

1.OLTP Transaction (MQ)

CEB Active-Active Deployment Model

22

Continuous Availability – Active-Active

Headquarter Gateway

(Route by BIN)

Encryption

Encryption

Front-end

App System

Core

Data

Core

Data

Syn

c

Syn

c

Beijing

Shanghai

Front-end App

System(Main)

Business Continuous Availability

Achieve Business Continuous Availability

by front end and mainframe active-active

Reliable Services

Synchronize application data based on

MQ reliable messaging, keep data

consistency in real time

Data Backup

Backup key business data through MQ

series

Data interchange in real time

The data centers could be located in long

distance

CEB Active-Active Logical model for OLTPSelf implemented replication service based on WebSphere MQ

for z/OS

Beijing Site Shanghai Site

MQ queue manager 1

send

VSAM

AOR

Transaction Publisher

VSAM

Transaction Replay

retrieve

MQ queue manager 2

AOR

Transaction Publisher

Transaction Replay

retrieve

send

Credit Card SystemCredit Card System

Workload Distributor

23

CEB Active-Active Data Flow

① Workload balancer switch workload to BJ or SH mainframe according to card type

② Card application update local data

③ Card application write transaction input ISO8583 data to MQ

④ MQ transfer ISO8583 data to peer site

⑤ User application read data from MQ

⑥ User application do identical update in remote site’s date sets.

AOR

MQ

AOR

MQ

Wrokload balancer

①

② ②

③③

④

④

⑤

⑤

⑥ ⑥

BJ MF SH MF

24

Planned Site Switch Over Procedure

Stop workload routing to BJ site

Waiting for SH site duplex as BJ site data

Workload re-rout to SH site

Reverse GM from site B to site A

25

Unplanned Site Switch Over Procedure

Stop workload routing to BJ site

Workload re-rout to SH site

Reverse GM from site B to site A

26

Characteristics of CEB’s Active-Active implementation

For business which has less complex master data with less

dependent database tables. For example, Credit Card business.

The synchronization applications need to be developed

according to your business and technical requirements, rather

than an out-of-box product.

27

References

WebSphere MQ V7.1 Information Center

InfoSphere Data Replication for DB2 for z/OS and WebSphere Message

Queue for z/OS: Performance Lessons, REDP-4947-00

GDPS Family An Introduction to Concepts and Facilities, SG24-6374-08

28

Questions?

We Value Your Feedback

Don’t forget to submit your Impact session and speaker

feedback! Your feedback is very important to us – we use it to

continually improve the conference.

Use the Conference Mobile App or the online Agenda Builder to

quickly submit your survey

• Navigate to “Surveys” to see a view of surveys for sessions

you’ve attended

30

Thank You

Legal Disclaimer

• © IBM Corporation 2014. All Rights Reserved.

• The information contained in this publication is provided for informational purposes only. While efforts were made to verify the completeness and accuracy of the information contained

in this publication, it is provided AS IS without warranty of any kind, express or implied. In addition, this information is based on IBM’s current product plans and strategy, which are

subject to change by IBM without notice. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, this publication or any other materials. Nothing

contained in this publication is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and

conditions of the applicable license agreement governing the use of IBM software.

• References in this presentation to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or

capabilities referenced in this presentation may change at any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to

future product or feature availability in any way. Nothing contained in these materials is intended to, nor shall have the effect of, stating or implying that any activities undertaken by

you will result in any specific sales, revenue growth or other results.

• If the text contains performance statistics or references to benchmarks, insert the following language; otherwise delete:

Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will

experience will vary depending upon many factors, including considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage

configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here.

• If the text includes any customer examples, please confirm we have prior written approval from such customer and insert the following language; otherwise delete:

All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual environmental costs

and performance characteristics may vary by customer.

• Please review text for proper trademark attribution of IBM products. At first use, each product name must be the full name and include appropriate trademark symbols (e.g., IBM

Lotus® Sametime® Unyte™). Subsequent references can drop “IBM” but should include the proper branding (e.g., Lotus Sametime Gateway, or WebSphere Application Server).

Please refer to http://www.ibm.com/legal/copytrade.shtml for guidance on which trademarks require the ® or ™ symbol. Do not use abbreviations for IBM product names in your

presentation. All product names must be used as adjectives rather than nouns. Please list all of the trademarks that you use in your presentation as follows; delete any not included in

your presentation. IBM, the IBM logo, Lotus, Lotus Notes, Notes, Domino, Quickr, Sametime, WebSphere, UC2, PartnerWorld and Lotusphere are trademarks of International

Business Machines Corporation in the United States, other countries, or both. Unyte is a trademark of WebDialogs, Inc., in the United States, other countries, or both.

• If you reference Adobe® in the text, please mark the first use and include the following; otherwise delete:

Adobe, the Adobe logo, PostScript, and the PostScript logo are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States, and/or other countries.

• If you reference Java™ in the text, please mark the first use and include the following; otherwise delete:

Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both.

• If you reference Microsoft® and/or Windows® in the text, please mark the first use and include the following, as applicable; otherwise delete:

Microsoft and Windows are trademarks of Microsoft Corporation in the United States, other countries, or both.

• If you reference Intel® and/or any of the following Intel products in the text, please mark the first use and include those that you use as follows; otherwise delete:

Intel, Intel Centrino, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and

other countries.

• If you reference UNIX® in the text, please mark the first use and include the following; otherwise delete:

UNIX is a registered trademark of The Open Group in the United States and other countries.

• If you reference Linux® in your presentation, please mark the first use and include the following; otherwise delete:

Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both. Other company, product, or service names may be trademarks or service marks of

others.

• If the text/graphics include screenshots, no actual IBM employee names may be used (even your own), if your screenshots include fictitious company names (e.g., Renovations, Zeta

Bank, Acme) please update and insert the following; otherwise delete: All references to [insert fictitious company name] refer to a fictitious company and are used for illustration

purposes only.

32