concepts, technologies, and best practices - an soa, bpm ... · leading soa and bpm technologies...

Concepts, Technologies, and Best Practices

Beth Gold-Bernstein & Gary So

Integration andSOA

next© 2006

http://ebizq.net/

�

About webMethodsGet There Faster.

webMethods provides total business integration to the world’s largest corporations and government agencies. webMethods flagship product suite, webMethods Fabric, leverages leading SOA and BPM technologies and methodologies to deliver rapid ROI to our 1,300 customers around the globe. With webMethods, customers can take a process-centric approach to their business problems, allowing them to re-use their existing IT assets, dramatically improve business process productivity and ROI, and rapidly create competitive advantage by making their business processes work harder for their company. webMethods (NASDAQ: WEBM) is headquartered in Fairfax, VA, USA and has offices throughout the United States, Europe, Asia Pacific and Japan.

For more information on our company and our products, we invite you to visit our website, www.webMethods.com

View our SOA webinar series – www.webMethods.com/Events/Web/GetRealwithSOALearn more about us – www.webMethods.com/AboutHear our vision – www.webMethods.com/VisionMeet our customers – www.webMethods.com/CustomersView our products – www.webMethods.com/Products

If you would like additional copies of the book, please email [email protected]

About ebizQ ebizQ – The Insiders Guide to SOA and Integration

ebizQ is the leading community of business and IT executives, managers, architects and developers focused on business integration and SOA. ebizQ’s analyst-centric multimedia content includes interactive training, webinars, and analysis of industry trends and best practices. You can access daily aggregation of in-house and industry-wide blogs, breaking news, feature articles and white papers.

Learn more by visiting www.ebizQ.net and gain access to the following great content!

Webinars: www.ebizQ.net/webinars/White Papers: www.ebizQ.net/white_papers/Online Curriculum: www.ebizQ.net/trainingcenterBlogs: www.ebizQ.net/blogsCIO audio: www.ebizQ.net/executive_corner/cioaudio/Buyers Guide: www.ebizQ.net/vendors

Bring the rest of your team up to speed. Attend our online companion course, a Manager’s Guide to Integration and SOA: www.ebizQ.net/managersguide or call 1-866-55-EBIZQ (1-866-55-32497)

http://www.webMethods.com

http://www.webMethods.com/Events/Web/GetRealwithSOA

http://www.webMethods.com/About

http://www.webMethods.com/Vision

http://www.webMethods.com/Customers

http://www.webMethods.com/Products

mailto://[email protected]

http://www.ebizQ.net/webinars/

http://www.ebizQ.net/white_papers/

http://www.ebizQ.net/trainingcenter

http://www.ebizQ.net/blogs

http://www.ebizQ.net/executive_corner/cioaudio/

http://www.ebizQ.net/vendors

http://www.ebizQ.net/managersguide

http://www.ebizq.net/mailabud/ebookform.html

�

Tables of Contents

Chapter 1 The Business Imperative for Integration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

Executive Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

Challenges to Business Agility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

The Mandate for Integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

Integration and SOA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

The Evolution of Integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

Integration: A Business Imperative . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

How to Use This Book . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

Chapter 2 Building the Business Case for Integration . . . . . . . . . . . . . . . . . . . . . . . . . . 15


Tangible and Intangible Benefits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

Areas for ROI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

Defining Costs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

Best Practices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

Chapter 3 Common Integration Scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32


Data Synchronization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

Enterprise Application Integration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

Mainframe Integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

B2B Integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

Single View of Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

Multichannel Access . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

Composite Applications. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

Business Process Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

Business Performance Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

Best Practices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52


�

Chapter 4 Guide to Integration Technologies. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54


Application Integration Services. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54

Information Integration Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

Interface Integration Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63

Composite Application Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

Process Integration Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67

Business Optimization Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70

Compliance and Industry Solutions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72

Management and Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73

Architecture and Development . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73

Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74

Chapter 5 Integration and SOA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75


Why SOA? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78

SOA Explained . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80

The Role of Web Services. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83

Enterprise-Class SOA. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89

Best Practices in SOA. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95

Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96

Chapter 6 Best Practices for Integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97


Common Pitfalls . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97

Taking a Strategic Approach to Integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99

Organizing for Success . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100

Responsibilities of the ICC. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106

Creating an ICC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109

Critical Success Factors. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111

Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113


�

Executive Overview

Sometime around 500 B.C., the Greek philosopher Heraclitus said, “Nothing endures but change.” While change may have been a constant since time immemorial, the

rate of change is accelerating far faster than ever before, and this is having a profound effect on business. Business cycles are shrinking rapidly. The way business was con-ducted even a decade ago is no longer acceptable if an organization wishes to remain competitive. Organizations have had to change how they interact with customers, how they manufacture goods, and how they are organized and managed.

Rapid changes are only possible when the organization itself is agile. The notion of an agile business has long captured the imagination of business executives. The agile business is able to embrace changes in market conditions, organizational structure, and the regulatory environment without missing a beat. An agile business empow-ers the management team to focus its collective acumen on delivering substantially increased value to its stakeholders.

The desire for greater business agility is making integration increasingly important. Agility is the combination of speed and adaptability: speed to bring new solutions to market more quickly, and adaptability to new business requirements and competitive pressures. This can only be achieved when business processes are able to be changed nimbly. And that in turn is possible only when the underlying IT systems are integrated flexibly to accommodate the speed of change required.

Integration also enables organizations to leverage existing IT investments to stream-line their processes for greater efficiency and productivity. IT infrastructures need to utilize existing applications and systems to the extent possible. This requires an inte-grated infrastructure along with end-to-end visibility of the business processes across disparate systems. It also requires an approach that supports the implementation of business solutions from existing components. This is the idea behind Service Oriented Architecture, or SOA. (See Chapter 5, Integration and SOA.)

The Business Imperative for Integration1


�

Chapter 1The Business Imperative for Integration

Challenges to Business Agility

Currently, business agility is constrained by a number of obstacles. These include inflex-ible applications that cannot be easily changed or enhanced; non-integrated stovepipe applications; inefficient business processes; a lack of visibility into business processes and operations; challenges brought about by mergers, acquisitions, and regulations; and a lack of alignment between an organization’s strategic objectives and daily op-erations. Integration technology can play a role in removing these obstacles. In fact, in today’s business climate, organizational growth may be inhibited without integration.

Integration technology is a key enabling factor in helping business and IT executives transform their organizations; get to market more quickly; respond faster to business opportunities, competitive pressures, and regulatory requirements; and differentiate how their organizations do business. Business responsiveness has become a function of an organization’s ability to rapidly marshal the underlying IT systems in alignment with business needs. This means leveraging existing assets while creating new busi-ness functionality.

The Mandate for Integration

One only needs to look at the history of computing and the evolution of business soft-ware to see why integration is a priority for most chief information officers (CIOs) today. In the beginning, back-office operations were run by mainframe systems. These sys-tems were built to optimize expensive computing resources, but not business agility. They were—and remain—difficult to change and brittle to boot: Changing one thing could easily break something else. The legacy systems still in place today generally run an organization’s core back-end operations in a robust and reliable way. However, they are usually batch systems and are unable to respond flexibly to business needs for real-time information or new functionality.

That unfulfilled business need gave rise to the first wave of distributed computing with the emergence of minicomputers. Department managers could purchase them with their budgets and select them based on their particular needs. Often, the availability of


�


a packaged application that met 50% to 80% of the department’s specific needs led to the introduction of these new platforms.

Unix, PCs, and client server software further reduced the cost of department systems, and they began to proliferate throughout organizations. Low-cost desktop productivity and development tools—such as Microsoft Excel, Access, and Visual Ba-sic—supported ad hoc solutions, which also proliferated and often became more stra-tegic than they were originally designed to be. Individual business units established their own computing facilities and application development capabilities, in effect set-ting up shadow IT organizations. This gave them the desired level of agility—at least at first—to respond more rapidly and independently to new business needs. However, it also led to islands of automation: hundreds of applications spread across the organiza-tion, many of them on desktops. These applications used organizational information and were in turn used to report on business operations. But they were not under orga-nizational IT management and were certainly not integrated consistently.

The emergence of the Internet and the associated rush to e-business further punctu-ated the need for organizations—and, by association, IT—to become more responsive to market dynamics. But with different departments launching their own initiatives and often duplicating one another’s work, the result was reduced visibility and control, along with reduced economies of scale.

The widespread adoption of distributed systems often meant that large organizations had multiple platforms running hundreds of applications that managed similar infor-mation through different portions of various business processes. Unfortunately, these applications were not designed to integrate with one another, so the organizations had to find ways to keep the information in synch across the systems. Rekeying the data—a means of last resort, but all too often the approach taken—was slow, resource-intensive, and prone to errors that could be costly to trace and resolve. So an easier way to integrate the disparate stand-alone systems was needed.

Another problem was that each packaged application was designed to focus on spe-cific department processes. Business agility requires the optimization of business pro-cesses end-to-end. This includes improving the process for initiators such as customers,


�


partners, and suppliers as well as people in different business roles, and developing systems that support different parts of the business.

While some emerging technologies look for a business problem to address, integration was a business problem long before it was a technology. Packaged software systems were far from being turnkey solutions. The integration costs of implementing the typi-cal enterprise resource planning (ERP) system could be three to five times the cost of the software. The problem was that the integration involved point-to-point hand-cod-ing. This required an understanding of the application program interfaces (APIs) of all the systems to be integrated and a high level of expertise. It also took a great deal of time and was inflexible to change. The number of interfaces rose exponentially with the number of systems being integrated. Such integration “spaghetti” was difficult to manage and change, and upgrading to a new version of any of the applications meant going through the process all over again.

This brings us to the present. Although the current state of affairs is largely the result of short-term business decisions, it is ultimately viewed as an IT problem. That’s because business change is intimately tied to the underlying IT systems’ flexibility—or lack thereof. As markets move quickly and new opportunities and competitors emerge, or-ganizations are increasingly challenged to close the gap between their business needs and the lack of flexibility in their IT infrastructures. Perhaps the biggest obstacle to closing this gap is that change has to be implemented while the core IT operations that support the business continue to function smoothly and seamlessly. The challenge is akin to upgrading the wings of a plane while it’s in flight. This is where Service Oriented Architecture enters the picture.


�


Case Study 1-1 Company: RBank

Company Description

RBank is a pseudonym for the retail banking subsidiary of one of the nation’s largest financial services companies. It provides investment management, retail and commercial banking, re-tirement, consumer finance, and investment banking products and services to individuals and organizations throughout the United States and, for certain businesses, internationally.

Business Challenges

RBank had more than 60 different customer response systems and applications, and all customer service requests had to be routed to specific lines of business—a process involving several steps—before the customer was directed to the appropriate representative. RBank also faced challenges complying with an increasing number of federal laws. The Patriot Act, for example, expanded anti-money-laundering requirements through provisions to deter illicit financial activities that support the financing of terrorism. RBank needed to ensure that it was submitting the appropriate customer information to its agency in a timely manner to meet the compliance requirements.

How Integration Addressed the Challenges

RBank’s enterprise architecture division adopted an SOA and the webMethods Fabric inte-gration platform to build applications faster and cheaper. RBank representatives now have comprehensive information on all customer interactions with the bank on one screen, making the customer experience much more positive. Using the integration solution, RBank employees can make changes to customer information, such as address updates, in one place and have them propagated and synchronized across all customer systems. RBank also used the new integration solution to automate previously manual processes, including customer loan ap-plications. Applications are now routed through the integration platform to RBank’s financial systems for processing, and decisions are returned to the customer in real-time.

The SOA approach and integration platform solution has also played a critical role in compli-ance efforts. RBank can now connect directly to its agency to transmit all the required cus-tomer information and manage the verification process in real-time.

Bottom-Line Business Benefits

The 360-degree customer view has enabled RBank to customize services to customers as well as to cross-sell and up-sell offerings, improving customer satisfaction and growing sales opportunities. By automating processes and eliminating error-prone manual steps, RBank has realized substantial cost savings.


10


Integration and SOA

At the core, SOA is about creating systems out of standard building blocks. The concept is not new. Many of the principles underlying SOA, such as isolating functionality to promote reusability, are long-standing best practices. In the past, however, the adop-tion of SOA was hindered by a lack of widely accepted standards for putting together these building blocks. Now, the pressing demands of business—which necessitate in-creased system flexibility and adaptability—have inspired nearly all organizations to align on a single set of standards for SOA: Web services.

While SOA involves much more than Web services, standardization is largely respon-sible for removing the barriers to SOA and fostering a future where systems can be more easily assembled and incrementally modified. Today, in the face of growing com-petitive pressure and the accelerating pace of business, organizations realize that they run a risk if they do not move toward SOA.

So how are SOA and integration related? In fact, they are highly complementary.

SOA is inherently about a distributed architecture, with systems that span computing platforms, data sources, and technologies. A distributed architecture requires integra-tion. By standardizing how systems interoperate, Web services simplify the task of in-tegration. Web services alone, however, do not suffice. Organizations need an evolu-tionary approach to SOA that incorporates legacy (non-Web-services-based) systems. Integration software provides the bridge between the legacy systems and SOA, allow-ing organizations to leverage existing software assets while managing their transition to SOA.

Integration solutions also contribute mature technologies—such as messaging, rout-ing, data translation and transformation, and event management—along with organi-zational disciplines that are necessary for full-fledged, enterprise SOA. (See Chapter 4, Guide to Integration Technologies.) Moreover, integration capabilities such as business process management (BPM) and business activity monitoring (BAM) allow organiza-tions to realize a higher level of business productivity from SOA by enabling the opti-mization of business processes and the alignment of strategic objectives with opera-tional actions. In short, integration should play a central role in any organization’s SOA strategy.


11


The Evolution of Integration

An organization’s IT history remains its present. Organizations seldom have the luxury

of starting with a clean slate. With the vast investments that organizations have made

in systems and infrastructures, CIOs have an overriding financial imperative to leverage

existing resources to support new business requirements. In addition, it is very risky to

rip out and replace complex systems that serve mission-critical business operations.

So new or modified business services need to be delivered rapidly without affecting

current operations. They also need to maximize the use of in-place assets, with all their

associated constraints.

Integration software has emerged as an effective and viable approach to bridging

organizational silos. But so far organizations have had only modest success in truly

leveraging their existing in-place application assets—whether legacy or packaged ap-

plications—to support new business initiatives. To successfully deliver business agility,

organizations must evolve their definition of integration and its application to their

business objectives. No longer only about integrating systems, the role of integration

now also encompasses quickly assembling new solutions for the business, and moni-

toring and optimizing an organization’s business processes and operations.

Integrate. To manage the full scope of the business and to increase efficiencies, organizations must integrate previously autonomous business processes and their underlying applications. Integration software on its own is insufficient. At best, it will only allow organizations to streamline their existing processes. Business agility requires the ability to extend existing IT assets as reusable ser-vices. This is the concept behind SOA. Not only does it enable organizations to leverage the investments they have already made, but it promotes reuse, with all the associated benefits of reduced cost and improved quality. The new world order, therefore, demands an SOA-based approach to integration.

Assemble. Putting in place an SOA-based integration foundation enables the rapid assembly of new solutions from the catalog of available services. This ability to decompose monolithic applications into reusable business services, and then to recompose them as needed, contributes directly to business agil-ity by enabling organizations to rapidly change and assemble new business processes to support changing business priorities. When combined with BPM

4

4


1�


to enable processes to be composed from existing building blocks, SOA allows operational changes to be delivered faster, cheaper, with a higher degree of quality and with a higher return on assets.

Monitor. SOA also makes it uniquely possible for organizations to monitor ev-ery aspect of their key business processes and gain insight into how they can optimize these business processes. By using the services within an SOA-based system as measurement points, organizations can gain unprecedented visibil-ity into the metrics—such as order processing times, average order amount, and so on—that reflect the operational and business performance of a pro-cess.

Optimize. Then, by comparing these real-time performance metrics against previously defined key performance indicators, organizations can create a feedback loop that facilitates iterative improvement of their processes—for example, by helping identify and reduce the bottlenecks in an order-to-cash workflow. Again, SOA plays a role by simplifying and speeding an organiza-tion’s ability to make the desired business process changes, or implement the necessary system solutions, to drive these improvements. This ability to iteratively improve the way the business operates ultimately results in greater business process productivity for the organization.

Integration: A Business Imperative

Business agility is becoming a strategic goal to organizations in every industry, and

in both the private and public sectors. The bottom line is that IT must be able to re-

spond more quickly to changing business needs, opportunities, and threats. It is a mat-

ter of survival. As discussed in this chapter, agility is enabled through an integration

infrastructure implemented with SOA principles. The glue that binds together all these

business goals and implementation strategies is integration. Organizations need to

create core competencies around integration and SOA, much the way they did with

databases and data centers in the 1980s and ’90s. Integration is nothing less than a

core requirement for enabling business agility.

4

4


1�


Case Study 1-2 Company: Electro

Company Description

Electro is a pseudonym for a leading global distributor of electronic components and comput-er products. With more than 100,000 customers and 300 suppliers, it markets, distributes, and adds value to a wide variety of electronic components, enterprise computing products, and embedded systems. The company is positioned between multiple tiers of customers—value-added resellers (VARs), contract manufacturers, and end users—and a large supplier base.Business Challenges

One of Electro’s major suppliers mandated that its software customers buy through the distribution channel. This forced multiple VARs to quickly choose a distribution partner for order fulfillment because they could no longer buy directly from the manufacturer. Most of the VARs used EDI for business, and they needed to select and migrate their current EDI ordering processes to a new partner within 60 days. Electro saw the opportunity to provide drop-in integration capability and capture the substantial revenue stream. To capitalize on the oppor-tunity, however, it needed to quickly put in place a solution that would connect the two ends of the supply chain by allowing each partner to leverage its existing EDI solution.


Using an out-of-the-box EDI solution from webMethods, Electro accelerated the creation of an EDI framework so potential VAR customers could connect for fulfillment. The company lever-aged its existing webMethods integration infrastructure, which contained many key integra-tion points and core services necessary to construct a solution rapidly. The EDI modules were installed into the infrastructure and utilized for document translation and data transformation. The resulting framework was reusable and enabled Electro to provide drop-in integration capability to the targeted VARs. Bottom-Line Business Benefits

Electro created more than $200 million in new, incremental revenue with a minimal invest-ment ($50,000). With the new EDI integration solution, it quickly added a number of key VARs to its customer list. In addition, the project yielded productivity gains of 30% to 45%. This included an estimated 60% of reuse from existing integration assets, which sped time-to-market. The solution enabled Electro to not only target new customers and their potential revenues but also create entirely new business models that otherwise could not have been addressed.


1�


How to Use This Book

This book is designed as a primer for those who need to understand integration and how to apply it in their organizations to achieve business goals. The book is organized as follows:

Chapter 2, Building the Business Case for Integration, more deeply defines the business benefits and return on investment that can be achieved through in-tegration. This chapter helps you build a business case for integration proj-ects.

Chapter 3, Common Integration Scenarios, helps you identify the business ini-tiatives that can be driven by integration technology.

Chapter 4, Guide to Integration Technologies, further explains the technolo-gies available and how they relate to different business solutions.

Chapter 5, Integration and SOA, defines the relationships between SOA and integration, and shows how to optimize your IT infrastructure to enable busi-ness agility.

Chapter 6, Best Practices for Integration, offers guidelines to ensure that enter-prise integration becomes a core competency in your organization.

Throughout the book, we have provided case studies that illustrate the benefits of in-tegration in specific situations. Although we have changed the companies’ names to maintain anonymity, they are actual organizations that have used integration products to achieve their goals.

In closing, remember that integration is not an end point. It is not a single technology that comes in a shrink-wrapped box, nor is it even a single project. As you read this book, it will become increasingly clear that integration is a journey. Success requires technology, process, and people. We sincerely hope that this book will be a valuable guide on your integration journey.

4

4

4

4

4


1�

Executive Overview

Many past investments in infrastructure technology have failed to deliver real business benefits and return on investment (ROI), making some organizations

leery of investing in integration software. However, integration is a key component in the creation of a more agile enterprise. It can help businesses streamline operations, provide better customer service through access to more complete information, and enable faster response to competitive threats. Progressive organizations are leverag-ing integration strategically to fundamentally differentiate their products and services and to gain competitive advantages that drive revenue growth. In IT, integration can reduce the cost and time required to implement new solutions while improving ef-ficiency. More intangible benefits include the ability to enhance the organization’s brand value by enabling modern, consumer-oriented solutions and a higher degree of service quality.

Business cases for integration should reflect a combination of tangible returns in hard dollars and more intangible returns that, although difficult to enumerate, offer sustain-able business value.

Organizations also need to understand that integration was a requirement long before packaged integration middleware solutions became available. Before the availability of off-the-shelf integration technology, implementing a software package with inter-faces to other applications, or combining systems because of a merger or acquisition, typically meant hard-wiring the systems together in a point-to-point fashion using hand-coded interfaces. With this type of integration, the number of interfaces involved rises exponentially for each additional system being integrated. Furthermore, the pro-prietary nature of each interface makes it impossible to leverage existing work when adding new systems. And, because they are hand-coded, the interfaces tend to be brittle and resistant to change. As a result, upgrading a software package can require integration costs several times the purchase price of the package. The net result is that

Building the Business Case for Integration2


1�

Chapter 2Building the Business Case for Integration

a significant percentage of IT organizations’ budgets are being consumed by the high ongoing maintenance and support costs for their home-grown integration solutions.

When making a buy vs. build decision for integration software, organizations need to factor in the total cost of ownership. In nearly all cases, off-the-shelf integration soft-ware solutions deliver a better ROI than internally developed alternatives. Off-the-shelf integration software is also typically more functional and robust, since the vendor can invest more R&D dollars into development and provide support for the software.

This chapter helps you to identify the benefits of integration and to build a business case and calculate the ROI you can expect from your investment in a packaged integra-tion solution.

Tangible and Intangible Benefits

Because of the complexity of integration and the mixture of tangible and intangible benefits, it is rarely possible to build an integration value case that is purely financially based. Instead, the overall business case for investing in an integration infrastructure is likely to have three parts:

Dollars representing the direct cost benefits of integration

Dollars representing the indirect benefits of integration

Soft or intangible factors that must be considered as additional weight in any judgment

Direct cost benefits from an integration solution represent actual dollars saved, such as reduced development costs for new projects, reduced maintenance costs, reduced errors and the cost of fixing errors, and reduced resource needs.

Assigning estimated values to indirect benefits is more challenging but equally es-sential. While indirect benefits may be difficult to quantify in the short term, they often provide the largest long-term benefits. The key here is to ensure that the affected busi-ness units agree with the assessed benefit value. For example, if improved integration

4

4

4


1�


delivers a more efficient just-in-time process for a manufacturing company, one ben-efit will be reduced inventory. On this basis, the inventory manager can provide a value estimate. It should be possible to validate this figure as the project is deployed.

The third part of the justification—soft or intangible benefits—is the most difficult to address, but it is nevertheless an important part of the consideration. Suppose the in-tegration project delivers a more responsive customer self-service system. One benefit should be improved customer satisfaction and, hence, customer loyalty. Though this may be a key organizational goal, assigning a dollar value to the benefit is difficult. Or suppose the integration project is part of fulfilling legislation, such as Sarbanes-Oxley requirements. Again, it is hard to place a dollar value on this, though noncompliance would have significant implications for the business.

Soft or intangible factors make evaluating integration projects an art as well as a sci-ence, in that a value judgment is required in addition to hard numbers. Nevertheless, decision makers can still obtain a meaningful broad-brush feel as to whether the in-vestment makes sense. And later refinement will make the case more accurate, even if some benefits remain intangible and subjective.

Figure 2-1. Examples of benefits to consider in an integration investment decision


1�


Areas for ROI

Integration is a key infrastructure technology for delivering a new breed of more agile enterprise applications, but it does present challenges for justifying investments. First, business investments are usually made on a project-by-project basis, while infrastruc-ture investments span multiple projects. It is often easier to justify investments across projects if an organization takes an enterprise approach to integration and the recom-mendations come from an Enterprise Architecture Group or Integration Competency Center (ICC) within the Architecture Group. To justify integration infrastructure invest-ments, look for both IT and business benefits, as well as benefits to the organization as a whole.

IT BenefITs

IT can benefit from integration through decreased costs and increased agility. Cost re-duction can be measured in real dollars. Increased agility is more difficult to quantify, but it can provide potentially much larger long-term benefits and ROI.

Direct Cost Benefits

The direct cost savings for IT through integration include reductions in three areas:

Development and implementation costs

Skills requirements

Operating costs

Integration software provides a single interface point to individual applications. Instead of programming to a different application program interface (API) for each application, integration platforms and Web services provide a standard interface to the integra-tion software that acts as a broker. This drastically reduces the number of interfaces required—and, thus, the number of interfaces that need to be developed, tested, and maintained. The savings can be calculated by first determining the cost of develop-ing interfaces for each application that needs to be integrated, then adding the cost of maintaining those interfaces, and finally comparing the total cost with the cost of purchasing the vendor-developed and maintained adapters.

4

4

4


1�


Case Study 2-1 Company: ScandAir

Company Description

ScandAir is a pseudonym for an Iceland-based airline that carries 1.5 million passengers each year on its fleet of Boeing 757 and 767 jets. ScandAir has sales offices and flight destinations across Europe and North America, and serves tourist and business traffic to Iceland as well as interna-tional traffic via its hub in Reykjavik.

Business Challenges

Internet bookings of airline tickets were rising, and ScandAir’s Web sites felt the impact. This evolving business channel had the potential to reduce costs and improve efficiencies, but it was having the opposite effect. Each online sale required an average of 11 minutes of administrative back-office support, negating any business benefits possible from e-commerce.

At the same time, ScandAir’s IT department was evaluating the need for an integration platform to help solve the increasingly complex web of enterprise applications that powered the company. ScandAir’s financial system, for example, interfaced with more than 30 applications. Managing this had become increasingly challenging. The company was looking for an area to test a new enterprise integration solution that would demonstrate a measurable ROI. The automation of its online airline ticketing process provided the ideal pilot project.


ScandAir wanted to use enterprise integration to deliver a fully automated e-ticketing system that would eliminate the time spent on its current manual processes. Using a local integration partner and components from the webMethods Fabric suite, ScandAir rolled out a successful initial pilot in less than three months that automated the back-office process associated with Internet bookings in the Reykjavik sales office.

After this project’s success, the effort quickly expanded to include all bookings out of Reykjavik. Then the model was implemented in each ScandAir sales region. The project included developing adapters for the various back-office systems, including the TPOS credit-card clearing system and the AMADEUS flight reservation system. These adapters were then used across the organization for other purposes, saving development time and effort.


Since ScandAir’s first implementation in 2003, more than 150,000 bookings have been completed. The automation has saved more than 27,500 hours of manual effort when compared with the previous processes. In addition, standardization has brought new efficiency gains in business pro-cesses and IT. ScandAir now processes its invoices electronically—something it did not originally foresee—and this has produced additional savings as a result of reduced postage and paper use and reduced storage space for the paper invoices.


�0


Integration software also reduces skills requirements because, typically, the integration software shields developers from having to deal with many underlying technicalities. An integration suite offers a standard approach that benefits not just the implemen-tation process but also operations, further reducing the requirement for special skills. This may eliminate the need to hire or contract with expensive specialist personnel. Additionally, since a vendor is now responsible for the integration mechanism be-tween components, it can be relied on to adapt to new technology advancements (for example, ensuing that connectivity to new versions of an application such as SAP or Siebel is maintained).

Finally, integration software can decrease operating costs by reducing transaction er-ror rates. It automates the transfer of information into systems and eliminates the need for rekeying, which reduces manual errors and thus offers significant ROI. Finding and fixing errors increases the cost of a transaction by orders of magnitude.

In fact, in ROI studies conducted by ebizQ, the reduction in error rates alone could quickly justify the cost of the system. For example, by using integration software, a major North American transportation company eliminated 70,000 to 80,000 hand-pro-cessed transactions, reducing the cost per order from $25 to $5. This amounted to a $1.6 million annual savings. In another example, a high-tech manufacturer using Ro-settaNet reported an 85% reduction in error rates and achieved a 100% ROI in the first year. And in 1999, 81% of Cisco’s orders were placed online with 98% to 99% accuracy, representing $11.7 billion in revenue annually. Organizations that can quantify error rates in their current systems processing are often able to assign a significant value to error reduction.

Indirect Benefits

The biggest indirect cost benefit for IT comes from the use of technology and solu-tions. For example, when Hurricane Katrina hit the Gulf Coast in August 2005, a North American insurance company wanted to make accommodations for its policyholders affected by the hurricane, such as providing a grace period on premiums. IT had already built a collection of Web services to access its mainframe systems in support of another large project. Now it was asked to quickly build Web pages for the custom site. Even


�1


though the high-level Web services developed for the other project were not suitable to support the new business requirements, a single member of the IT team was able to quickly—within a day or two—recombine lower-level worker services to create the necessary Web services for the Katrina site. This impressed the business managers and was recognized as a Service Oriented Architecture (SOA) success story up to the CIO level.

Soft Benefits

The soft benefits of integration for IT include increased responsiveness to the business and an increased perception of IT as a business enabler. New business processes can be deployed more quickly because of the reduced effort and the reusability offered by integration. Further, as new technologies such as Web services take hold, a coherent integration architecture delivers an enhanced level of system flexibility, enabling these technologies to be embraced and exploited faster and cheaper. And, of course, the use of vendor-supported integration technology means that responding to new technol-ogy challenges is the responsibility of the vendor rather than the IT department.

BusIness BenefITs

The business benefits of integration include enhanced process efficiency and produc-tivity, increased operational effectiveness, and improved competitive performance. En-hanced efficiency and productivity often translate into direct cost savings. Increased operational effectiveness often provides many indirect benefits. Improved competitive performance can be the most valuable benefit to the organization, but also the most difficult to quantify.

Direct Cost Benefits

Integration software improves efficiency and productivity in a number of ways. First, in-tegration can increase levels of automation and reduce human intervention. Examples include reducing order times by removing manual order reentry at different stages of the processing cycle, cutting the percentage of failed funds transfer requests due to input errors, and improving the speed of information sharing between different


��


departments. Cost reductions would result from decreased staffing levels due to the increased level of automation or the use of a more productive process. In addition, pro-cess efficiency may also allow a reduction of inventory through just-in-time processes that rely on a high degree of automation. This becomes even more feasible when the process efficiency is extended to encompass suppliers and distributors, and it can have the side effect of reducing procurement costs as well.

When business process management (BPM) functionality is layered on top of basic integration capabilities, process efficiency and effectiveness are increased, and this can have very quantifiable bottom-line results. In the 1950s, Dr. W. Edward Deming, a math-ematician and statistician, showed that variations created at every step in a process can cause defects, which are expensive and time-consuming to remediate. Deming’s work with Japanese manufacturers was directly responsible for the low-cost, high-quality goods that competed favorably against American manufactured goods.

Deming’s work gave rise to a number of initiatives to manage and measure the effi-ciency and effectiveness of business, including Total Quality Management (TQM), Bal-anced Scorecard, Six Sigma, and ISO 9001. Organizations that adopted these practices realized measurable results. In fact, some integration software products on the market today enable organizations to apply these quality management and measurement practices to their business processes in the act of integrating them.

Indirect Benefits

Operational effectiveness creates more indirect benefits. Integration can help make a business more effective by improving business intelligence and its dissemination. For example, the ability to pool information about a particular prospect from different parts of the business may make cross-selling or up-selling possible. Or the ability to provide comprehensive feedback on product performance directly from customers to Engineering may enable the development of new processes to improve product qual-ity. Assessing the ROI depends on the specifics of the situation; for example, benefits might be realized by reducing the cost of sales or increasing sales volumes.


��


Case Study 2-2 Company: Annuate, Inc.

Company Description

Annuate, Inc. is a pseudonym for an Australian super-annuation fund administrator. The company man-ages 5 million member accounts and more than 340,000 employer accounts. It provides administrative services such as bulk contribution processing, fund take-ons, benefit processing, insurance and claims management, member communications, and the management of member inquiries.

Business Challenges

For several years, Annuate had been vying for a major new account, the Australian Retirement Fund (ARF), with more than 600,000 members. Although Annuate had always been competitive in the ten-der process, it learned that its business process applications did not meet ARF’s requirements. Annuate identified its technology architecture as the main reason for losing the business. And it realized that to be competitive it would need to build a world-class technology infrastructure founded on a solid integration strategy that would enable new systems to be deployed while leveraging existing legacy systems.


Annuate deployed webMethods Fabric to integrate with a new client’s applications and facilitate real-time dissemination of information across the enterprise. This new enterprise integration infrastructure allowed it to win the ARF account and to merge 600,000 ARF records in the course of a six-month project that established the funds’ records on the core administration system, provided data confir-mation and business rule coding, and made the other applications work off its Oracle-based member administration system.

As part of Annuate’s integration efforts, it adopted an SOA platform that enabled the company to reuse many of the services it created. Annuate can now expose Web services through the integration of applications such as Web sites and internal administration systems, and can repurpose the components of its existing legacy systems, further leveraging existing technology.

The integration also allows Annuate’s clients to view their portfolios online, get daily estimates of their account balances, view regular e-newsletters, and learn about investment changes. These online tools complement the information provided on member statements. The real-time updates of information ensure accurate information to customers and across the enterprise.


The ARF win was a direct outcome of Annuate’s clear enterprise integration strategy. The implementa-tion of the enterprise integration solution has allowed a number of back-end systems to interact and provide a complete administration service for the client. As a result, Annuate has already seen a 30% to 40% reuse of interfaces and services created, as well as reduced maintenance costs. The entire solu-tion also enables IT to better support Annuate’s business strategy.


��


Soft Benefits

Being able to serve customers’ needs faster and cheaper improves the business’s chances of beating the competition. In addition, the ability for process changes to be implemented faster at the operations level, through underlying IT mechanisms that are more flexible to change due to integration, enables business operations to respond to competitive action more quickly. Once an integration infrastructure is in place, it reduces the cost of implementing change. It could also lead to increased revenue by enabling a business unit to lead with new differentiating initiatives. These competitive gains can be evaluated in monetary terms by assessing the likely market share increase that results from operational efficiency and speed of reaction.

OrganIzaTIOnal BenefITs

Finally, there are the returns at the organizational level. Of course, the two areas already described—IT benefits and business benefits—already have an effect on the overall organizational performance. But this section concentrates on areas that apply more specifically to the organization as a whole, reflecting the brand value across all aspects of the business. These benefits include:

Compliance with government and industry regulations

Customer satisfaction

Exploitation of the organizational knowledge base

Access to new opportunities

Most organizational returns can be categorized as indirect or soft benefits. Benefits may result even when integration itself is secondary to the underlying business require-ment. For example, many organizations are finding unexpected benefits from their compliance initiatives. The ability to integrate different information sources improves the understanding of organizational operations and procedures. Moreover, particu-larly with some of the more recent forms of legislation, such as Sarbanes-Oxley and Basel II, it enables organizations to build sophisticated audit trails of their operations. This has the beneficial side effect of providing organizations with a better window into

4

4

4

4


��


how their business runs. In reality, any integration project that plays a key role in meet-ing compliance mandates is likely to be justified on a nonfinancial basis (indeed, it is often a case of organizational survival).

Integration plays a significant role in customer satisfaction. It can provide a complete, organization-wide single view of the customer in real (or nearly real) time. This picture may include current product sales, product experiences, reported issues, outstanding deliveries, and many other facets of customer-related information. This capability al-lows the organization to provide a much higher level of service to the customer, for example through its call centers or help desks. Customer satisfaction leads to customer loyalty, which many organizations can relate to financial performance, for example by comparing the expense of securing new customers with the cost of retaining existing ones. Regardless of how meaningfully the benefit can be quantified, customer satisfac-tion is an important benefit of integration.

Integration can also bring together knowledge from all of an organization’s business units, making it available for sharing and thereby increasing the overall value of the organizational knowledge base. This can have paybacks in improved organizational efficiency, more rational organizational structures, and a more effective workforce. For example, providing employee portals that offer information about organizational pro-cesses and procedures, such as health and safety, may simultaneously reduce educa-tion costs for new employees, help improve employee morale, and achieve compliance with legal requirements.

Finally, integration can link together the entire value chain, both within the organiza-tion and outside to partners and even customers to enable to new opportunities. This wide-reaching integration makes it possible to introduce new business products or initiatives. In fact, many organizations are using integration strategically to offer new services to new markets.

Defining Costs

To evaluate ROI, the benefits that accrue from the integration project must be balanced


��


against the costs incurred. Broadly, IT costs tend to be grouped around the provision and support of the IT changes required to implement the desired operational changes. In contrast, business unit and organizational costs tend to be expressed in terms of the changes required to adapt to the new operational processes and procedures. It is important to remember that both are relevant, although the costs from the business side of the organization are harder to evaluate. Organizations that forget the impact to business personnel generally suffer as a result of the oversight.

IT COsTs

IT costs resulting from an integration project generally fall into three categories:

Infrastructure costs

People costs

Services

Infrastructure costs include the expected software license and maintenance costs. These cover the integration solution selected, any new application packages, and any other new pieces of infrastructure software required. It may also be necessary to ac-quire hardware to handle the additional capacity generated by the solution or to pro-vide local processing power as needed. While the initial infrastructure costs may be high, they are easier to justify when amortized across multiple projects. This is another good reason to take an enterprise approach to integration.

Clearly, a wide range of people costs must be included in any evaluation. These span the traditional range of project development, encompassing planning, design, devel-opment, testing, deployment, and support. At the planning and design levels, it is im-portant to have both IT and business skills, since integration projects deal with both aspects (for example, business processes have to be defined or modified to produce the required business return, and this must be explained to the IT team). In addition, business domain knowledge may be required to help IT understand the business stan-dards, such as RosettaNet, that need to be taken into account in the solution.

4

4

4


��


Case Study 2-3 Company: DNA Corp.

Company Description

DNA Corp. is a pseudonym for a life sciences company that specializes in research technolo-gies, diagnostic products, and targeted therapeutics. Pharmaceutical and biotechnology com-panies use its products to discover and develop new drugs more effectively. DNA Corp. has an installed base of approximately 180,000 instrument systems in nearly 100 countries.

Business Challenges

DNA Corp. builds customized assays for its customers. This is a complicated process that involves performing detailed analyses of DNA sequences. Traditionally, a customer uploaded a file containing the sequence to DNA Corp.’s e-commerce Web site. The order was then routed via FTP through DNA Corp.’s back-office systems, which included 13 enterprise systems from order creation through accounts receivable. This process was error-prone, and recovering from any problems that developed was labor-intensive. The development time for each new assay could take 12 to 16 weeks. DNA Corp. needed to streamline the process for building custom-ized assays to improve its competitive position and reduce costly errors.


DNA Corp. wanted to separate the process of managing and accessing the information about the assays from its order entry system in SAP. To streamline the process, it adopted an SOA-based enterprise integration solution from webMethods. The new system allows customers to define assay requirements through a Web front end, which then passes an XML message to the design engine. The engine attempts to produce an assay design to match customer requirements. The customer can validate the design and then choose to have the assay built based on optimizing cost, accuracy, repeatability, or other factors. These additional parameters determine the process sequences used. Externalizing the assay information from SAP provides greater flexibility and reuse, and reduces the overall time to build a customized assay.


The new enterprise integration solution has had a direct impact on DNA Corp.’s bottom line. By integrating the 13 back-end systems involved, it reduced the development time for a new assay type from 16 weeks to two weeks. This enabled DNA Corp. to reduce its order fulfillment errors by 20% and decrease overall order time from three weeks to less than five days.


��


The design phase needs to consider not just the technical architecture and approach

but also the operational processes and workflows necessary to support the solution.

Development resources are required to build the interfaces to the components be-

ing integrated, to modify the relevant IT infrastructure, and to supply essential mecha-

nisms, such as audit and problem determination trails. The integration software itself

has to be configured for the operating environment.

The test phase may require specific hardware and software, for instance to validate

performance requirements and to provide a thorough quality assessment of the new

software implementation. Personnel are required to carry out the testing and to resolve

and rework problems as required. Finally, deployment and support costs must include

items such as help desk and operations staffing and training, quite possibly with indi-

viduals who have business as well as technical skills, to ensure successful operations.

Throughout the range of project activities, it may also be necessary to consider third-

party service costs. Some organizations may decide to utilize systems integration ex-

perts to provide valuable guidance in the planning and design phases of the project.

Parts of the project may also deal with a specialized area of technology knowledge,

and in these cases it may be more financially viable to contract for the skills rather

than retrain existing IT staff. Examples include retaining someone to integrate a par-

ticular software package or someone skilled in the particular integration technology

selected.

BusIness ImpaCTs and COsTs

The three categories most likely to be taken into account in the area of business costs are:

Resources to support project implementation

Education and training of end users

Operational risks

4

4

4


��


The first category has already been described in the previous section. Because integra-tion projects span business and technical disciplines, resources to support planning, design, implementation, and deployment activities are required. These resources are necessary to ensure not just that the right solution has been built but that the new solution is deployed in a controlled fashion to minimize disruption and business risk.

The education and training of the end-user community is another important cost to consider. If the integration project is simply automating some back-room operations, then the end-user experience and procedures may remain unchanged. But if new or changed business processes are being implemented, retraining is an essential element to be considered. Additionally, if the integration project spans business partners as well as internal departments, the partners may also need education.

The area of operational risks covers aspects such as the loss of productivity during the retraining and changeover to the new process, and potential service disruptions due to technical problems or user errors experienced as the solution becomes fully estab-lished. Since it is difficult to determine the impact of these issues and their associated cost, decision makers may simply want to note them as part of their overall evaluation criteria.

Best Practices

The following guidelines will assist you in determining the potential ROI for your inte-gration project.

Know what you are measuring. Obviously, when measuring the success of any project, you must first know what you are measuring. Set your objectives for the ROI audit. Be-fore you can measure improvements, you need to know what your current processes cost. Take baseline measurements before you begin the project. Measure the initial performance levels of the processes under consideration for change or optimization to build a baseline for future comparison. If you have already defined critical success factors, you can do a baseline measurement of them. We recommend budgeting one to two weeks for the initial estimate. Then determine how often you want to measure your progress against your goals. We recommend a quarterly or semiannual cycle.


�0


Get executive commitment. Identify an ROI audit sponsor. This person is usually the senior manager of the business unit affected by the implementation. Ideally, he or she should be able to sell the audit findings throughout the organization and be willing to defend the logic, assumptions, and results confidently.

Focus on improving business processes. Ultimately, improving business processes will yield the highest return on investment. To begin, identify some initial processes that are the low-hanging fruit—ripe for automation and integration.

Start small and lay the foundation for future enhancements. Nothing succeeds like suc-cess. Make sure the first projects have a high probability of succeeding. Keep the scope relatively small but still pertinent to the business.

Avoid analysis paralysis when doing the ROI study. Set a number of hours or days com-mitted to ROI analysis, and stick to it.

Build a credible business case for integration. Do your homework to build the analysis from the ground up, and its credibility will show. Avoid forcing numbers. Be careful not to work from the goals down to the supporting numbers. When working on the analysis, many assumptions and variables may change numerous times throughout the process, and these often will ripple to affect other parts of the analysis. Be sure to check, double-check, and triple-check all the calculations. One error could shoot your credibility in the entire analysis.

The checklist on the following page will help you build your business case for investing in an integrated infrastructure.


�1



��

Executive Overview

Most midsize to large corporations and government organizations have a myriad of computing platforms, business applications, databases, and other technolo-

gies distributed throughout their organizations. These systems were designed to meet the needs of specific parts of the organization and were not intended to be integrated with other systems. Business requirements have changed, however, and integration has become a fundamental necessity for any business process that involves more than one system or the coordination of work across different groups of employees.

Addressing a particular business need may involve several different integration re-quirements. For example, implementing a supply chain management solution might involve connecting electronically to suppliers and customers, managing the manual processing of order exceptions, synchronizing order details among different internal systems, and providing a single view of customer transactions and details across differ-ent parts of the organization.

It is possible to break down almost all business integration requirements—even those involving the most complex business processes—into a few basic integration scenarios (or “use cases,” to use a software engineering term). These scenarios can be thought of as building blocks for implementing more complex, integrated business solutions.

The basic integration scenarios are:

Synchronizing data among different databases and applications

Automating transactions among different internal systems—commonly re-ferred to as enterprise application integration (EAI)

Integrating mainframe-based applications and data with distributed systems

Connecting businesses electronically and automating transactions with part-ners, suppliers, and customers—known as business-to-business (B2B) integra-tion

1.

2.

3.

4.

Common Integration Scenarios3


��

Chapter 3Common Integration Scenarios

Providing a single view of information that is distributed across different sys-tems and databases

Extending access to systems over the Web and via other channels, such as mo-bile devices and interactive voice response units

Repurposing IT assets by turning existing functions into components that are then assembled into new composite applications

Automating and managing business processes and workflows

Using integration to optimize business performance

Just as the integration scenarios vary, so do the technologies available to implement them. In this chapter, we discuss the integration scenarios and mention some of the technologies. Then, in Chapter 4, we discuss the integration technologies in more de-tail. Together, these two chapters provide a guide to choosing the right integration ap-proaches and technologies for an organization’s specific business requirements.

Data Synchronization

An organization usually has multiple systems and platforms, and similar information is inevitably kept in many of them. Information about customers, orders, and products can span customer management, order entry, fulfillment, warehousing, and billing sys-tems. Keeping all this information synchronized is essential to the smooth execution of many business processes.

Figure 3-1. Sharing data among multiple systems

5.

6.

7.

8.

9.


��


Many organizations synchronize data by rekeying information manually into systems.

However, this method has a high error rate, and fixing an error increases a transaction’s

cost by magnitudes. Automating data transfer provides direct cost benefits by reduc-

ing the personnel costs of both rekeying data and fixing errors. Some organizations

have found that error reduction alone delivers a sizable enough return on investment

(ROI) to justify an automated data integration solution.

Once an organization decides to automate data synchronization, it must choose which

approach to take: transferring flat files, performing database-level updates, or using

an application program interface (API) or application adapter. Because information re-

sides in disparate systems and platforms, data mapping and transformation must be

done so the information is delivered in the right format for the target platform. Data

synchronization patterns, from least to most complex, include one-to-one (synchroniz-

ing data between two systems), one-to-many (synchronizing data from one system

across two or more systems), and many-to-many (synchronizing data from multiple

systems across multiple systems). Data synchronization can also be one-way or bidirec-

tional. If it is bidirectional, a mechanism is needed to prevent an endless loop in which

an update to System A triggers an update to System B, which triggers another update

to System A, and so on.

Enterprise Application Integration

Whereas data synchronization addresses the need to duplicate data across different

systems, application integration addresses the need to link different applications in-

volved in the same business process—for example, an order management process that

spans customer relationship management (CRM) and enterprise resource planning

(ERP) packages. Mergers and acquisitions also create the need for application integra-

tion, for example, to integrate financial and HR systems.


��


Figure 3-2. Exchanging transactions among applications

Application integration is at the core of a variety of business problems. It can be a com-ponent of B2B applications, composite applications, and business process automation (as described later in the chapter).

Application integration can sometimes be achieved at the database level—for exam-ple, extracting an order from System A’s database and inserting it directly into Sys-tem B’s database. In these cases, the programmer needs to understand the underlying structure and meaning of the data and ensure that data consistency rules are adhered to rigorously so the applications receive data in the format they require.

More typically, application integration occurs at the transaction level, such as during the creation of an order. The transaction—usually in the form of a message sent from one system to another—is typically processed through an API or an application adapter that simplifies connectivity with the application. This level of integration enforces the application’s business logic. However, data translation and transformation are still nec-essary to ensure that transactions are properly formatted for the receiving systems.

Critical components of application integration include transactional integrity, secu-rity, the ability to trace what is happening, and exception-handling mechanisms. While


��


exceptions often must be processed manually, they should be managed as part of the end-to-end automated and integrated process.

Application integration technologies include adapters, message brokers, EAI suites, en-terprise service buses (ESBs), and application routers. Similar in concept to network routers, application routers combine hardware and software to address simple integra-tion needs.

Mainframe Integration

Mainframe integration can be part of various integration scenarios, including data syn-chronization, application integration, B2B, composite applications, and multichannel access. However, the unique intricacies of the mainframe environment demand ad-ditional considerations.

In general, it is difficult to gain access to mainframe applications from the outside be-cause they were not designed for this. Mainframe operating environments are tightly controlled and do not lend themselves to external access beyond the originally pre-scribed mechanisms, such as via terminals or reports.

Figure 3-3. Accessing mainframe systems and data

Mainframe access methods are broadly classified as invasive or noninvasive. Invasive methods require changes to the mainframe code. They are difficult to implement but


��


may be worth the trouble when the integration needs to support a high transaction volume. Noninvasive methods require few or small changes to the mainframe. In some cases, no additional software is required. Options for noninvasive mainframe integra-tion include the following:

Screen scraping, which replaces a human operator with a computer program that interacts with the mainframe application through a terminal interface (for example, 3270).

Report-based interfaces, which use the output from reports to trigger data-oriented integration processes.

Direct data access via gateways to mainframe-resident databases such as DB2. These products are sometimes referred to as SQL gateways.

Transaction-level integration to CICS or IMS. This can be noninvasive if the mainframe application offers that interface, or minimally invasive if imple-mented through a message-based interface.

In the long run, the best solution for mainframe integration is “wrappering” existing mainframe transactions—that is, using integration technology to noninvasively make existing transactions accessible via different protocols—into reusable Web services that can be accessed from any source. Though it has an initial development cost, wrap-pering enables greater reusability and flexibility.

Each approach to mainframe integration requires different technologies. The best ap-proach depends on the characteristics of the mainframe application, the implementa-tion time frame, and whether the integration is part of a tactical or a strategic imple-mentation. Other factors may include the available skill sets, budget, and available tool sets.

Market consolidation has left only a handful of specialized legacy integration vendors. Many integration platform vendors partner with a specialist on OEM legacy integra-tion technologies to offer mainframe integration capabilities as part of their broader suites. Whether an organization is implementing a legacy extension solution as a tacti-cal solution or building a legacy integration service layer as part of a Service Oriented Architecture (SOA) strategy, it needs to understand how this capability fits into the overall architecture.

4

4

4

4


��


B2B Integration

Automating transactions with partners, suppliers, and customers is commonly referred to as business-to- business, or B2B, integration. Organizations require B2B integration when they are attempting to streamline the supply chain, connect to an electronic exchange, or comply with industry and regulatory mandates that require electronic communications to government or business entities outside the organization.

Figure 3-4. Integrating business processes with trading partners

Processing electronic transactions with business partners can be 10 to 100 times less expensive than processing the same transactions manually. Organizations typically have used a value-added network (VAN) solution to implement electronic interchange of purchase orders and other electronic documents, usually based on electronic data interchange (EDI) formats and standards. However, VAN- and EDI-based solutions are expensive to implement. This is due to the complexity of processing EDI and VAN charges, which are based on the volume of data transacted. As a result, EDI is used mostly by larger organizations—and by smaller organizations only when they need to do business with larger organizations.

With the advent of Extensible Markup Language (XML) based integration technologies that utilize the Internet and open standards rather than expensive proprietary technol-ogies, alternatives have emerged. EDI is not going away, but EDI systems are evolving


��


to integrate with newer standards-based technologies. Together, the new integration technologies are making it possible for many more organizations to participate in elec-tronic commerce.

Integration with systems completely outside the organization’s control pose some challenging integration requirements. Additional security is necessary to authenticate partners and ensure that the transactions are legal and cannot be repudiated. Orga-nizations also need a way to manage service level agreements with each partner and provide different connectivity options for small, midsize, and large organizations. B2B integration also requires support for different data formats, including EDI, XML, and custom flat files, as well as standard industry protocols, such as RosettaNet and the Health Insurance Portability and Accountability Act (HIPAA).

B2B is also often associated with industry exchanges such Covisint, which started as an automobile industry exchange and has expanded into healthcare, and the World Wide Retail Exchange (WWRE), which was set up by 17 international retailers to automate retail supply chain processes. These exchanges offer supply chain services that enable suppliers and partners to conduct electronic business using the Internet rather than a proprietary network as the transport layer. They allow all partners to integrate with the exchange, rather than integrating point-to-point with all their partners and suppliers. This saves considerable time and money.

Exchanges may also provide additional value-added services. For example, Covisint of-fers a Web-based tool to coordinate and manage problem-solving processes between customers and suppliers. WWRE offers a collaborative planning tool, a logistics tool, and other facilities that enable on-line trading. Ariba, another B2B exchange, provides buyers and suppliers with a number of services for conducting electronic business.

Today, B2B integration technologies are generally available as part of a broader integra-tion platform or in a hosted environment through integration service providers, such as GXS. Technologies for implementing B2B solutions include basic application capa-bilities, partner management capabilities (which address the need to manage process-ing rules and service levels on a partner-by-partner basis), and capabilities for handling B2B-specific security requirements, such as firewall friendliness and the encryption of documents exchanged over the Internet.


�0


Case Study 3-1 Company: IT Corp.

Company Description

IT Corp. is a pseudonym for a Fortune 50 information technology company with more than 150,000 employees doing business in more than 170 countries. The company offers products to consumers, small and midsize businesses, enterprises, and the public sector. It invests heav-ily in R&D, producing more than 10 patents a day worldwide.

Business Challenges

IT Corp. has been dedicated to automating the computer industry supply chain and B2B trans-actions. Traditionally, all supply chain transactions, from forecasts to invoices, were handled via paper and email. This process was very inefficient and translated into high costs, increased turnaround times, and data errors from manual processing. IT Corp. identified the supply chain process as business-critical and wanted to create a Web-based channel that would enable supply chain transactions between IT Corp.’s customers and suppliers through a B2B Web site.


Using the business integration solution from webMethods, IT Corp. was able to link its Baan ERP system with an internally developed Active Server Pages Web application. This enabled the company to use all its existing business logic that resided in Baan for the new Web appli-cation. With the Web-based supply chain system, business partners can now conduct all trans-actions on the B2B Web site. Customers can place new orders, request changes to existing or-ders, view and print order acknowledgments and shipping details, and submit order forecasts to IT Corp. to streamline the procurement process. Suppliers can easily view and acknowledge orders, provide updated shipment details, and access procurement forecasts to manage sup-plies. The new system processes between 10,000 and 15,000 transactions each month.


Delivered in just three months, the Web-based B2B portal has become an integral part of IT Corp.’s supply chain system and supports a growing percentage of the company’s transactions. The automated process has streamlined and simplified the purchasing process for the com-pany’s suppliers and customers. By completing simple online transactions, customers receive immediate responses and acknowledgments of process completion, and this has increased customer satisfaction. In addition, the automated system has eliminated time-consuming and error-prone manual processes. The real-time nature of data flows across the system provides accurate information at all times. The new system has effectively saved thousands of man-hours of work for IT Corp. and its business partners.


�1


Because most organizations do business with many types of partners, they need differ-ent types of on-ramps, or methods of connecting to partners and suppliers. The larg-est partners may have full-blown B2B implementations, including full integration with their back-end systems. Some midsize partners may just need to send and receive XML messages. And smaller partners may require browser-based connectivity where they manually submit transactions or check on status. Ultimately, the goal is to make it fast and easy to connect, because the ROI for B2B implementations is directly related to the percentage of partners participating.

Single View of Information

With the proliferation of applications and databases in an organization, information about customers, suppliers, employees, products, orders, and other business entities becomes spread out, and it is difficult to get a single, consistent view. Integration soft-ware can help organizations aggregate information from various systems to create a single view of any business entity. This capability is especially important for business solutions used in situations such as mergers and acquisitions, CRM, or integrated risk management.

Figure 3-5. Combining information into a single view


��


One way to gain a single view is to physically aggregate data into one place. Many

organizations have created data warehouses and data marts to accomplish this. How-

ever, data marts and warehouses are generally updated at predefined intervals, such

as nightly or weekly, so they may lack the most current information. An alternative

method is the distributed query, which involves keeping the information in the source

systems and extracting the information as needed into a combined view, as if all the

information were coming from a single source. This ensures that the information pre-

sented is up-to-date and the most current available. The tradeoff with a distributed

query is that it typically takes longer to get a response than it does when making a

request to a single database.

An important dimension of single-view integration is the need to understand both the

underlying data structures of the various source systems and the semantic meaning of

the data in each system so they can be reconciled into the single view (whether in a

data warehouse or in real-time). It is necessary to determine, for example, that the Cus-

tomer field in System A is the same as Name in System B, CustName in System C, and

Client in System D, and to map the transformation of data as it is extracted from the

sources. A major ambition for data integration is to capture the semantic meaning of

information within systems in metadata, which allows systems to be integrated more

easily without having to create each mapping manually.

A number of new solutions are coming to market that enable a real-time single view

of information. These include enterprise information integration (EII) solutions, which

have proprietary technologies and algorithms for aggregating information in real-

time, such as distributed query optimization, indexing, and caching. Some solutions

include semantic integration capabilities. For physical-aggregation-style approaches,

traditional data integration technologies can be used to source data from—and popu-

late data in—the databases involved.


��


Case Study 3-2 Company: FoodCo

Company Description

FoodCo is a pseudonym for one of the leading food and beverage companies in the United States. It is the largest processor and distributor of milk and other dairy products, and oper-ates more than 120 plants in the United States and Spain.

Business Challenges

Over the years, FoodCo had acquired several new businesses, which operate as independent business units. Each acquisition brought more SKUs or UPC codes, and managing them had become increasingly challenging. Data for identical products was inconsistent among busi-ness units, and finding and validating information that customers needed took a long time. In addition, maintaining product information was a complex and error-prone process, especially when items had multiple levels of descriptions, such as formulas for ice cream flavors. FoodCo realized that it needed to create an enterprise-wide product data management process to ensure data consistency across business units and guarantee the delivery of information to the right place at the right time.


Using the webMethods integration solution, FoodCo created a single master-item system that aggregates data from multiple back-end systems and routes it to users for action. The new system enables more than 100 employees across 32 business units to create, edit, validate, and manage product data. The system also generates alerts to resolve exceptions that could lead to data inconsistency.

Customers now see FoodCo as a single vendor with a consistent product catalog. They can make purchases in a timely manner because they receive notification beforehand and can react more efficiently to what the company sends them.


The new master-item system enables FoodCo to manage more than 35,000 SKUs in nine categories and to add more than 200 new items each year. For its 360,000 customers and partners, the system has reduced the time for new product introductions from five days to one. Because the enterprise integration platform synchronizes the data within the existing ERP systems, FoodCo can leverage its investment in internally developed systems and meet cus-tomer mandates on synchronized items. The webMethods platform has also enabled FoodCo to implement a complete standards-based EDIINT/AS2 solution between VAN and non-VAN customers, resulting in a five-figure savings on annual VAN charges.


��


Multichannel Access

The past few years have seen a proliferation of devices through which end users and consumers can access computing services. No longer tethered to an application run-ning on a desktop computer—or, before that, a mainframe terminal—people can now use Web browsers, personal digital assistants (PDAs), mobile phones, interactive voice response units, and other special-purpose devices to receive and send information. As organizations start creating reusable business services, they inevitably want to make these services available through different channels.

The expansion of access channels represents an opportunity for organizations to bet-ter serve their employees, customers, suppliers, and partners anytime and anywhere, but it also presents a significant IT challenge. The associated technologies are expand-ing and changing rapidly, and organizations will increasingly need to provide services across a variety of devices based on a variety of technology environments.

Rather than altering existing applications to allow for each type of access channel, or developing solutions that are hard-wired to a specific type of device, several emerging technologies and standards initiatives allow multichannel access to different back-end systems.

Figure 3-6. Using different end-user channels to access back-end systems


��


Case Study 3-3 Company: Comco, Inc.

Company Description

Comco, Inc. is a pseudonym for a U.S.-based Fortune 100 global communications company that provides mobility products and solutions for the home, automobile, and workplace. The company’s 2004 sales exceeded $30 billion.

Business Challenges

Comco’s mobile device business unit was losing significant margin because of duty fees for phones shipped from its U.S. plants to Latin America. The company recognized that divesting the manufacturing of these phones to a regional manufacturer would reduce its duty fee costs in the Latin America market by several million dollars each year. Comco’s major challenge was to define an automated process that would transfer the manufacturing and distribution of mobile phones to the third-party contractor without reducing on-time shipments to Comco’s customers. Another challenge was transforming Comco’s existing ERP system into two sepa-rate but seamless systems, one located at the new contractor’s plant. This would involve a cutover process with minimal interruptions for each company.


Comco had made a formal commitment to RosettaNet, an industry standard for B2B com-merce. It wanted to build on the RosettaNet work and create an enterprise integration solu-tion that would provide touchless integration from Comco’s back-end Oracle ERP system to the third-party contractor’s back-end SAP system. Comco implemented a webMethods-based solution that enabled the new system to automatically exchange a variety of information about engineering, purchase orders, and other product information. The system was also enhanced with business activity monitoring (BAM) capabilities that send alerts to business managers when process exceptions occur, allowing them to react quickly and proactively.


The new system has enabled Comco to relocate its manufacturing to Latin America and avoid several million dollars in duties annually. The company has also reduced its operating costs by close to $1 million each year through the elimination of manual processes, effectively reducing the number of people required to support the business while maintaining on-time shipments to customers. To date, the touchless system has enabled more than 1.2 million mobile phones to be shipped to Latin American customers with no direct contact by Comco. In addition, order tracking and other information is provided in real-time, giving the company added bandwidth to focus on new business opportunities. This is a significant competitive advantage for Comco.


��


A midtier server is the least invasive and most flexible way to provide interface ser-vices to multiple types of front-end devices. The midtier server renders the interface to fit the display characteristics of each device. This way, the back end does not need to change the way it sends the information for each type of device.

For accessing back-end systems, Web services are a good choice because they enable access from a broad range of clients-side systems, including Web servers, Java, C#, and mobile devices. Additionally, the use of XML and style sheets enables information to be displayed in different formats for different devices without requiring changes to the underlying data.

Few organizations will be able to ignore the demands for multichannel access from multiple constituencies, and for many it will soon become a requirement. Organiza-tions therefore need to create a strategy and an architecture for multichannel access and integration.

Composite Applications

Organizations seeking to rapidly implement new business solutions soon realize that they cannot pursue a rip-and-replace strategy. Starting fresh with each new system simply costs too much and is too disruptive to the organization. Unfortunately, many applications hinder rather than enable business change. Their rules, processes, data definitions, and user interfaces are embedded in compiled code that requires program-ming skills and detailed background knowledge to modify. To meet business needs, organizations must leverage the systems they have in place but gain flexibility in how the systems can be repurposed. Composite applications are the answer.

Composite applications are assembled (or composed) from existing components rath-er than developed from scratch. They also incorporate functionality from a variety of underlying systems.

This approach to building new systems aligns closely with Service Oriented Architec-ture. SOA is an architectural approach to creating reusable building-block components


��


as services that interoperate through standardized Web services interfaces and that can be combined quickly and easily to address new business process needs. A compos-ite application consists of functionality drawn from several sources within an SOA. The components may be individual Web services, selected functions from other applica-tions, or entire systems whose business functions have been enabled as Web services (often legacy systems).

Figure 3-7. Composing new applications from existing underlying business services

An example of a composite application is a self-service portal for consumers or busi-ness partners. Here, a number of existing processes supported by IT services might be combined within the portal to provide the end user with one-stop access to a range of the organization’s products and services. Without an SOA approach, this type of composition would require considerable application integration, programming, and testing. With an SOA, however, the composite application can be assembled easily,


��


regardless of the location or the technology of the different services being drawn to-gether. This allows IT to respond faster to business demands, keeps costs to a minimum, and powers imaginative new ways to address end users’ needs more effectively.

One caveat must be mentioned here. Whereas a monolithic application typically pro-vides tools to monitor status (for example, to view the status of an order), a composite application offers no such built-in monitors because the status crosses multiple ser-vices on potentially different platforms. Therefore, for monitoring and other functions such as policy enforcement, composite application implementations need an infra-structure that exists outside the services themselves.

The technologies that enable composite applications include traditional application development platforms, integration suites, Web services, and Web service repositories that enforce governance policies, provide security, and enable reuse. The services from which composite applications are composed can be created by wrappering existing application functions or by building new ones from scratch. As such, a tool set is need-ed to catalog, discover, and orchestrate services into a new application, and to deploy and manage the result. Additionally, a run-time deployment infrastructure is needed to connect all the components in a scalable and reliable way.

Business Process Management

As integration technologies and solutions have matured, the scope and complexity of the problems that organizations are looking to solve have increased. Beyond ensuring data consistency, linking a few applications together, or exchanging documents elec-tronically with a trading partner, organizations are increasingly looking to integration software to streamline entire business processes.

Business processes are the lifeblood of an organization. Whether related to customer service, product development, or service delivery, an organization’s business processes define its ability to compete successfully in the market. Process-level integration offers the richest business benefits. Organizations can increase competitiveness through dif-ferentiated business processes, and IT can better keep pace with changing business requirements. So, while eliminating manual data entry and automating transactions can yield direct and measurable cost benefits, automating and optimizing business


��


processes holds the greatest potential for reducing labor costs and cycle times; im-proving employee, customer, and partner satisfaction; and increasing business agility

Figure 3-8. Coordinating end-to-end business processes across systems and people.

By nature, business processes are often complex and disjointed. Different organization-al entities may be responsible for different parts of the process, with no single entity responsible for the end-to-end process. In addition, business processes often include both automated and human processes. Some parts of the process may be performed automatically by IT systems, whereas other parts may involve people performing spe-cific tasks. In fact, typical enterprise business processes run the gamut from fully au-tomated to semi-automated to entirely manual. Although organizations may want to automate as much of the process as possible, people remain an important part of the equation. Individuals are needed to assess exceptions, apply judgment and experience, and handle other situations requiring context that is not easily encoded in a system.

The term business process management (BPM) generally refers to the set of activities that organizations perform to either manage or optimize their business processes and adapt them to new organizational needs. Because these activities are usually aided by software tools, BPM also refers to the category of integration software that provides these capabilities.


�0


Since business processes are typically a combination of automated and manual tasks, BPM solutions should support both types of processes. Human processes require a user interface. Typical human processes involve delegation and organizational consid-erations, such as the different roles of managers and employees. For example, manag-ers need the ability to assign task to employees with certain skill sets, manage their employees’ workloads, balance the workloads when assigning or reassigning tasks, and provide management oversight and intervention for manual activities.

BPM solutions typically provide graphical tools for modeling business processes, a plat-form for automating and executing these processes, and facilities to monitor them. They may also provide simulation capabilities to test and optimize business processes.

Business Performance Optimization

Henry David Thoreau said, “In the long run, we only hit what we aim at.” When organiza-tions have tools that deliver real-time visibility into—and performance measurements of—business processes, they can focus not only on what goods and services they are delivering but also on how they are delivering those goods and services. This focus is necessary for business performance optimization.

Business performance optimization is a combination of methodology, best practices, and technology infrastructure. There are two levels of business optimization. As dis-cussed above, business process management solutions enable process-level optimi-zation—in essence, streamlining the execution of business processes. Business per-formance management solutions monitor relevant business events, which may cross multiple processes, and relate them to key performance indicators (KPIs).

Both levels of business optimization are required. In fact, it can be argued that business optimization should be performed at all levels of the organization, from the operational IT level, through line-of-business managers, up to senior executives. All require visibility into the KPIs for which they are responsible. Alignment across all organizational levels is required for overall business optimization. Otherwise, an organization risks optimiz-ing one process while suboptimizing at an enterprise level.


�1


Business activity monitoring (BAM) is an emerging technology for enabling business performance optimization. In 2002, Gartner coined the term BAM and defined it as “the concept of providing real-time access to critical business performance indicators, along with the supporting information to improve the speed and effectiveness of busi-ness operations.” Whereas BPM solutions provide visibility across individual business processes (or process-level management), BAM solutions allow organizations to corre-late events from multiple processes and relate them to measures that are meaningful to the enterprise. In doing so, BAM provides a “big picture” of business performance and helps align business strategy with operational execution.

BAM solutions provide business dashboards that correlate real-time business events with KPIs. Some BAM solutions include predictive capabilities and can identify emerg-ing patterns that may influence a KPI, giving managers the information necessary to proactively handle emerging problems rather than simply react to erupting crises.

Figure 3-9. Improving business performance with BAM technology


��


BAM is often associated with business intelligence (BI) technology. Consider them complementary sides of a coin, with BAM providing the real-time operational view into business performance and BI the view for historical analytical purposes. Some BAM vendors offer the ability to integrate with and feed into existing data warehouses, al-lowing organizations to use their BI tool of preference.

Best Practices

This chapter illustrates how integration can be applied to a broad range of business requirements. Chapter 4 provides more detail on the underlying technologies used to implement the scenarios described in this chapter.

It is important to understand that midsize and large organizations have a variety of business requirements and therefore require a variety of integration technologies. Be-low are some best practices to increase the odds that an organization’s integration initiatives will succeed in the long run.

Plan strategically; implement tactically. Each of the integration scenarios presented in this chapter could be approached as a tactical implementation or as a necessary technical task within an overall strategic implementation. While such an approach might yield short-term benefits for “low-hanging fruit,” in the long run it will increase redundancy, software license costs, skills training, and management and maintenance costs. In contrast, an enterprise architecture approach to integration will enable long-term business agility. It will make different integration technologies available to differ-ent projects as needed, ensure that the technologies are integrated with one another, and decrease the time and cost to implement new solutions.

Create an Integration Competency Center (ICC). As an organization strives to de-velop core competencies for managing integration, an ICC will help it more efficiently build and maintain the complex integrations that its business demands. The ICC is re-sponsible for the development and deployment standards that will help facilitate reuse and interoperability as well support the implementation, maintenance, and continual evolution of the organization’s integration platform. (See Chapter 6 for a description of the ICC’s function and roles.)


��


Practice continuous improvement. Business agility is a journey, not a destination. Or-ganizations need to evolve their infrastructures, business services, and processes to optimize the business at all levels. Technology is just part of the solution. Real-time visibility into processes and KPIs is only useful if someone is paying attention and prac-ticing continuous improvement based on the enhanced information. An organization’s success depends largely on what it does with the technology. Organizations need to build a culture of continuous improvement and business optimization. This requires that executives, line-of-business managers, and business operatives are all aligned along the same business goals, and that a reward structure is in place to promote the desired outcomes. The good news is that integration technology can also help solve this problem through management and operational dashboards.


��

Executive Overview

As mentioned in Chapter 3, different business scenarios warrant the use of different integration technologies and infrastructure services. Therefore, organizations will

inevitably acquire a collection of technologies to meet their diverse business needs. To maximize agility and reuse, organizations need to understand how the range of inte-gration solutions available today relate in the overall enterprise architecture.

The ebizQ Integration Roadmap can help organizations match their business require-ments with the appropriate integration technologies. Becoming familiar with the Roadmap is a good way to understand both the integration landscape and the types of business problems addressed by specific technologies.

Application Integration Services

Application integration services offer the basic features and capabilities required to integrate and connect applications reliably and securely. These services include:

Connectivity to the applications. Connectivity is generally achieved using adapters, which enable a standard application connectivity approach and eliminate the need to program against proprietary application program inter-faces (APIs). If Web services interfaces are available, application connectivity may be achieved without an adapter. Application connectivity may also be achieved at the database level using a database adapter or a direct SQL inter-face.

Data transport and routing capabilities to move information reliably and se-curely between systems. Application integration solutions typically use a mes-sage-based approach to moving data and support a variety of topologies that link systems together in a scalable and flexible manner.

Data transformation and mapping capabilities. These enable data to be con-verted to and from the required target representations.

4

4

4

Guide to Integration Technologies4


��

Chapter 4Guide to Integration Technologies

Infrastructure services to implement the application integration scenarios de-scribed in the previous chapter. These include support for transactions, appli-cation-level security, and exception management services.

Figure 4-1. ebizQ Integration Roadmap

The core application integration capabilities often play a role in other integration ser-vices, such as information integration, interface integration, composite applications, process integration, and business optimization with business activity monitoring (BAM). Without the underlying application integration services, these other integration services require point-to-point connectivity with back-end applications, which limits agility and reuse.

Technologies that provide application integration services include message brokers, integration servers, and enterprise service buses (ESBs). In addition, connecting with mainframe systems and integrating the systems of different organizations in a busi-ness-to-business (B2B) scenario have special considerations and, consequently, require additional technology solutions.

4


��


message BrOkers

The original application integration technology, message brokers provide the basic connectivity needed for application integration, including messaging, data transla-tion and transformation, and intelligent message parsing and routing. Message broker products support different styles of messaging—such as request/reply for sending a message to an application and receiving a reply, and publish/subscribe for broadcast-ing information and events to a number of applications. An important function of the message broker is to provide a secure and reliable communications layer—for exam-ple, to ensure the guaranteed delivery of a message from one application to another. Message brokers usually have a hub-and-spoke architecture, with all processing done at the central broker. More advanced vendor solutions offer distributed processing op-tions and multiple hubs to improve scalability and availability.

InTegraTIOn servers

As application integration requirements have become more complex, the basic mes-sage broker-centric architecture has evolved to provide more functionality at the in-tegration end points, such as supporting more sophisticated application processing rules, handling B2B standards like RosettaNet, and performing protocol translation (for example, layering a Web services interface around an existing application interface). These additional capabilities are typically combined into an integration server. Inte-gration servers can be used in combination with message brokers to create a scalable integration backbone, with multiple brokers connecting multiple integration servers. They can also be used alone for smaller point-to-point integration projects.

InTegraTIOn plaTfOrms

Since introducing their products in the mid-1990s, enterprise application integration (EAI) vendors have evolved their message brokers, integration servers, and adapter li-braries into integration platforms, which include additional capabilities and services to address different integration scenarios and provide more direct business value. These platforms, also called integration suites, generally include full B2B capabilities


��


and some level of business process management (BPM) and workflow capability. They may also include business activity monitoring features, a portal, mainframe integration capabilities, and other combinations of the integration technologies. These additional capabilities are discussed later in this chapter.

esBs

The enterprise service bus is an emerging middleware category that is typically viewed as a key enabling technology for Service Oriented Architecture (SOA). Just as previous generations of distributed systems depended on middleware such as RPC, CORBA, and transaction monitors, SOA relies on ESBs to help bind the services together.

ESBs serve as intermediaries between services providers and service consumers to avoid point-to-point connections. In this role, ESBs can fulfill requirements such as in-telligent routing, transformation, load balancing, and security while also coordinating the flow of service calls. Conceptually, ESBs provide much the same functionality in an SOA as an integration platform does in an application integration scenario. The biggest difference is that ESBs are generally based on Web services standards.

Some vendors originally positioned ESBs as simpler alternatives to full-blown integra-tion platforms. In reality, the lines have blurred as ESB vendors incorporate functional-ity similar to traditional integration platforms and suites.

adapTers

In a typical integration scenario, messages are routed between the various applica-tions being integrated. Once a message arrives at an application, it must be processed, usually by making updates or changes to the application or its database. The options for accessing the application include writing a program that uses the API, using a pre-built adapter, or using a Web services interface.

Many commercial applications have APIs, which enforce the necessary business rules, transaction logic, and data checks for programs that interface with the application. Ac-cessing the application via the API is thus preferred over integrating directly at the


��


database level. The downside to APIs is that they differ for every application, so the developer needs to know how to code to each application’s unique APIs. This increases the time and cost for integrating systems and decreases agility.

Adapters are a desirable alternative to API-level programming. They simplify the inte-gration effort by eliminating the need for application-specific interface development. Adapters provide a common set of function calls for different back-end applications. On the one side, an adapter exposes a standard interface to the developer; on the oth-er side, the adapter deals internally with the task of translating these interface calls to the specific APIs of a supported application. The developer needs only learn how to code the standard adapter interface rather than each application’s interface. Some vendors reduce the development effort further by offering graphical tools for config-uring adapters.

There are two types of adapters. Intelligent adapters can perform data translation and transformation at the source or target, thereby distributing the processing load and increasing the solution’s overall scalability. Standard (non-intelligent) adapters provide only basic connectivity services, thus requiring a message broker or integration server for translation and transformation.

Off-the-shelf adapters are not always available for every application that an organi-zation might want to integrate. So vendors frequently provide a development kit or library for building custom, one-off adapters that interface with the application while providing the niceties of a commercial adapter, such as graphical configuration or built-in logging and error handling.

WeB servICes

Web services are an attractive integration alternative because they provide a standard interface protocol that eliminates the need for adapters. However, Web services only replace adapters, not the rest of the application integration services. For example, they do not perform functions such as data translation and transformation, which remain an important feature of the overall integration solution.


��


B2B InTegraTIOn

Additional requirements exist when integrating with applications beyond the firewall.

These requirements include:

Support for a variety of electronic commerce standards and formats, including Electronic Data Interchange (EDI) and newer XML-based standards such as RosettaNet, as well as support for custom document and flat-file formats.

Support for connectivity to proprietary communications networks and over the Internet using protocols such as FTP, HTTP, and Web services.

More stringent security than what is typical for internal application integra-tion. This includes the need to manage data confidentiality and integrity and the identity of the parties involved.

Management of configuration settings and service levels on a partner-by-partner basis.

B2B integration solutions usually include a variety of connectivity options—sometimes

called on-ramps—for different types of partners. The challenge of B2B commerce is to

provide a range of access technologies to meet the needs of all partners and suppli-

ers. For larger partners, it might make sense to fully automate the B2B transactions,

including integrating with the back-end systems on both sides of the firewall. Smaller

partners with minimal IT resources require simpler solutions, such as partner portals

for inputting information or just sending XML messages. Tools that enable organiza-

tions to connect faster, such as integration and process templates, accelerate the time

to value.

A major evolution of B2B integration is in the area of EDI. Electronic Data Interchange

encompasses a set of standard protocols for conducting structured interorganization

exchanges, such as making purchases or initiating loan requests. EDI transactions are

typically conducted through a value-added network (VAN) provider. These VANs are

essentially private networks for conducting B2B transactions.

4

4

4

4


�0


The explosive use of the Internet has led to the creation of a new standard—EDIINT—that enables EDI transactions to be integrated with Internet transactions using XML. EDIINT has several variations, which are based on the transfer protocol used. EDIINT ASI uses SMTP, while EDIINT AS2 uses HTTP and HTTPS for securely exchanging EDI files over the Internet. Some integration solutions on the market today combine EDI and XML standards support in the same framework and tool set, allowing a single solution to be used for all B2B-related integration requirements.

B2B solutions also demand different—and in many cases greater—levels of security than does internal application integration. Authentication services are required to con-firm the user’s or organization’s identity. Authorization services are required to specify a user group’s levels of access to various applications and services. And nonrepudiation services are required to ensure the validity and enforceability of electronic contracts. The type of nonrepudiation needed depends on the type of application. Nonrepudi-ated communication sessions may suffice for simple data exchanges, but nonrepudiat-ed transactions are generally required when the solution includes more complex B2B transaction processing.

B2B integration exchanges and service providers have become viable alternatives for essentially outsourcing the B2B solution. Each partner needs only connect with the ex-change rather than manage the details of connecting with every partner. This reduces an organization’s burden when connecting to multiple partners with different technology infrastructures.

maInframe InTegraTIOn

Mainframe systems usually do not have exposed APIs and therefore present unique application integration challenges. There are several ways to integrate with these legacy systems. Options include screen scraping or report scraping, integrating with the legacy database, integrating at the transaction level through specialized gateways (for example, for IBM CICS or IMS transactions), wrappering legacy code as a service with a standardized interface, such as a Web service, or front-ending (that is, creating a message-oriented interface to an application), which requires some changes to the


�1


application. Specialized legacy integration solutions are often used in combination with other integration software to connect the mainframe systems with applications running on distributed platforms.

Information Integration Services

Information integration services consolidate and integrate structured and unstruc-tured information from multiple sources, and manage it at an enterprise level to pro-mote data consistency. The value of the information in an enterprise system is in direct proportion to its quality and consistency with other systems. When data is managed as an important enterprise resource, it provides greater value.

The key to managing distributed information effectively is metadata, which is informa-tion about data represented independently from the underlying systems. Metadata plays an important role in enabling interoperability among systems because it allows data to be understood and exchanged consistently across the systems. Enterprise metadata repositories that provide real-time data services enable reuse and agility.

The next leap forward in data management services and integration productivity will come from semantic integration. Semantic integration captures the meaning and con-text of data within source systems in a metadata repository. Every integration project requires an understanding of the underlying information in the systems. It is crucial to know, for example, that CustName in one system is the same as Company in another system. Generally, this knowledge is put into proprietary mapping tools. By captur-ing it as semantic metadata, organizations can maximize reuse and ultimately enable systems to automate the rules for translation and transformation. This also increases return on investment (ROI) because the effort and cost of discovering the semantic meaning of data within systems can be more readily reused across multiple projects.

Enterprise information integration (EII), enterprise content management (ECM), and metadata repositories are the newer technologies that focus on information integra-tion. Additionally, organizations will continue to use traditional data integration ap-proaches—such as data warehouses and extract, transform, and load (ETL) tools—to deliver the full set of information integration services.


��


eTl

Extract, transfer, and load technology was developed for data warehousing implemen-tations. ETL tools operated on large data sets in batch mode, extracting the informa-tion from systems on a nightly or weekly basis, cleansing it, transforming it into the desired target format, and then loading it into the data warehouse or data mart. ETL technology is still used today for applications where it is preferable to physically move the information to create a unified view and where real-time updates are not neces-sary, such as for historical analytics. However, the newer information integration tech-nologies discussed below offer the option of leaving the data in the source systems and retrieving it in real-time as needed.

eII

An emerging integration market sector, enterprise information integration provides an aggregated view of data. EII solutions can provide the equivalent of a virtual data ware-house or midtier data store for real-time reporting or fast implementation of a portal solution. EII tools have technologies for distributed query optimization, indexing, and caching that enable real-time information access and aggregation. Graphical views make it easier for programmers and business managers to access the information they need without knowing the underlying data structures of the source systems. Whereas data warehousing requires that the information be moved to a central database, EII solutions access the data from the source systems themselves. Emerging EII solutions are also incorporating semantic integration capabilities.

eCm

Enterprise content management provides integrated access to and management of unstructured data, such as documents and images. ECM solutions can provide work-flow and process management as well as integration capabilities. ECM services are re-quired when the business process is primarily focused on documents being passed around the organization, or when unstructured content needs to be part of an overall business process integration solution.


��


Interface Integration Services

Interface integration services provide access from a variety of front-end devices to a

variety of back-end systems. The technologies that provide interface integration ser-

vices include portals (for delivering the user interface via a browser) and mobile inte-

gration gateways (for enabling the use of mobile devices).

pOrTals

Portals provide an integrated user interface to multiple back-end systems. Many EAI

solutions focus on automating transactions across systems. But a portal solution inte-

grates information into a browser for a particular purpose or set of users, providing a

single touch point for the delivery of application services.

Various types of portals exist. Some—such as enterprise information portals, which pro-

vide information on corporate policies or news—are informational only. That is, they

aggregate information and functionality from multiple systems, but the information

flows only one way: to the portal. Transactional portals provide greater interoperability;

for example, enabling customers to place and track orders, or allowing employees to

enter expenses or manage their 401k’s. There are also management portals for tracking

business processes or key performance indicators (KPIs), and operational management

portals for monitoring system-level performance.

Portal technology often includes an environment for developing the portal applica-

tion. This application can be composed of multiple portlets, which are pluggable user

interface components. For example, there may be a log-in portlet that manages log-ins

to multiple back-end services. Portals may also include personalization features that al-

low users to customize the look and feel of the portal application—for example, choos-

ing what types of information to display, such as local weather or a customized stock

ticker. Although portal technology provides a way to develop an integrated user inter-

face, back-end integration is still required. This can be done point-to-point or using the

application integration technologies discussed above.


��


Several important standards address portals. Web Services for Remote Portlets (WSRP) defines a set of standard interfaces for creating portlets that can be accessed remotely using Web services protocols. This allows applications to consume portlet components without having to write unique code for interacting with each component. And JSR 168 is a Java API for portlets that enables content sources and application front ends to be aggregated. It also addresses how security and personalization are handled.

mOBIle InTegraTIOn gaTeWays

Mobile devices—including cell phones, PDAs, and specialized mobile units such as package tracking systems—all have different screen sizes, resolutions, and interface requirements. These mobile devices are proliferating rapidly. Today, people commonly use their cell phones to receive email, surf the Web, and conduct other aspects of busi-ness.

IT departments are coming under increasing pressure to provide enterprise application services to a variety of mobile devices. Mobile applications are likely to require back-end integration to present information from multiple sources. In addition, because mobile devices operate on inherently unreliable and insecure networks, management and security need to be part of the integration solution. And mobile applications of-ten must accommodate occasionally connected users, who need to work offline and then synchronize with source systems and reconcile the offline transactions across the systems.

Mobile integration solutions usually include a specialized server that provides the transformation services for each target device, along with security. This server inte-grates with back-end systems either directly (point-to-point) or through an integra-tion platform. The mobile integration server typically uses message queues and logs to guarantee that messages will be delivered once, and only once, even in the event of a system failure. To support occasional users, it controls message delivery by organiz-ing messages into transactional groups, and determines when messages are delivered based on the type of network connection. To address the insecurity of mobile commu-nications, the ability to encrypt data is important.


��


Composite Application Services

The term “composite application” refers to an application assembled from components

that may originate from various underlying systems. Composite applications enable

new business solutions to be created by flexibly combining new and existing appli-

cation functionality. This allows organizations to deliver systems faster than building

them from scratch. The benefits of a composite application approach are maximized

when organizations create reusable business services—which then serve as building

blocks for a composite application—as part of a Service Oriented Architecture. As a

result, composite applications are typically associated with SOA and Web services.

Composite application services are the technical infrastructure services that support

the development, deployment, and management of composite applications. These in-

clude the ability to:

Develop new services (typically based on Web services standards)

Create service abstractions by combining several low-level services into a more useful, reusable high-level business service

Design and model the behavior of the composite application, including how information and messages flow across services, and how services need to be orchestrated in support of a business process

Discover services or browse from a registry of available services

Manage policies and service level agreements to engender trust and enable the reuse of services

Deploy the resulting solution into a managed run-time environment

The technologies most commonly associated with composite applications include

Web services, service orchestration capabilities, and service registries for governance.

Software vendors are introducing composite application platforms to offer a collection

of these technologies and services in an integrated suite.

4

4

4

4

4

4


��


WeB servICes

Web services and their related standards have emerged as the de facto building blocks for composite applications. Web services define a standardized interface (Web Services Description Language, or WSDL), a standardized communication protocol (Simple Ob-ject Access Protocol, or SOAP), a standardized repository for registering and discover-ing Web services (Universal Description, Discovery, and Integration, known as UDDI), and standardized message encoding using XML. These standards enable a Web service to reside anywhere and be accessed from everywhere, making Web services well suit-ed to the role of providing aggregated functionality within a composite application and being the standard for interapplication interfaces. As a result, many organizations today base their application development and integration efforts on Web services.

OrChesTraTIOn

To create a composite application from a set of Web services, it is necessary to define the flow of control across services. Doing this in programming code reduces the flex-ibility to change the application, so an externalized approach to service orchestration is preferable. This latter approach basically allows developers to modify the way ser-vices are linked together without changing the services themselves. Business Process Execution Language (BPEL), a vendor-led initiative, is gaining mind share and market share as the orchestration standard. A number of software vendors now support BPEL, making it possible to share process definitions more easily among tools.

servICe regIsTrIes fOr gOvernanCe

UDDI is the registry standard for dynamically discovering and invoking Web servic-es. UDDI was originally created to manage the cataloging of public Web services, but registries are also gaining ground within the enterprise. Organizations that embrace SOA will experience a proliferation of services—and versions of services. To reap the benefits of reusability, organizations must manage these services effectively through-out their life cycle. Beyond basic UDDI support, commercial registry products typically incorporate some level of Web services management capability to make the registry more useful within the enterprise.


��


In addition, some vendors use the registry as the basis for policy and governance man-agement in their solutions so that policies—such as service level agreements—can be enforced at run-time. This approach allows the policies to be enforced across all ap-plications that utilize the service, rather than having each application implement the policy independently. That means policies can be set at an enterprise level. As compos-ite applications grow in number, registries and service governance will become critical aspects of the integration infrastructure.

COmpOsITe applICaTIOn plaTfOrms

The development paradigm of composite applications involves assembling existing components, or services, into a new system. However, it will often be necessary to de-velop new services and to integrate composite applications with other enterprise sys-tems. Therefore, both the integration vendors and the application server vendors are beginning to position their platforms as—and evolve their offerings into—composite application platforms. Naturally, the integration vendors take an integration-centric view, while the application server vendors focus on the development aspects. Never-theless, a common set of composite application platform characteristics is emerging. These characteristics include custom development capabilities, integration capabili-ties, process orchestration, and user interface functionality.

Depending on the vendor, the composite application platform may be based on an ap-plication server that also includes integration and orchestration capabilities, or it may have an integration platform as its foundation, with process management capabilities and a registry for managing services. Because it is an emerging technology segment, the offerings are extremely varied.

Process Integration Services

Process integration services enable organizations to automate and optimize business processes that cross application and organizational boundaries. This helps organiza-tions reduce operating costs, gain real-time visibility into end-to-end business process-es, and increase business agility.


��


While underlying integration technologies and composite application development can help make IT more productive and agile, the optimization of business processes can deliver competitive advantage and significant ROI. As mentioned earlier, in the 1950s, Dr. W. Edward Deming proved that improvements in managing business pro-cesses lead to improvements in overall business and a significant competitive advan-tage.

A process-driven approach to integration also improves the alignment between IT and the organization. It starts with the development of business process models that busi-nesspeople can understand and review. These models depict a shared understanding of the end-to-end business process. Using process integration technology, the models can then be automated. This significantly reduces the chance of misinterpreting the business requirement or getting the process wrong. Since processes can span depart-ments, business units, and organizations, no single individual may own or even un-derstand the end-to-end process. Therefore, process integration tools need to foster collaboration on a process model and an iterative approach to its design and imple-mentation.

An organization’s business processes may be automated, manual, or collaborative. Fre-quently, an end-to-end business process includes both automated and manual pro-cesses. From an integration perspective, each type of process has different require-ments and therefore calls for different process technologies. The technologies that deliver process-level integration include business process management for handling automated (and sometimes manual) processes, workflow management for manual processes, and groupware or collaboration platforms. Some vendors offer solutions that combine these capabilities into a single product offering. Some also offer process simulation and analytics to provide feedback for optimizing processes.

Bpm

The business process management solutions on the market today represent a range of process integration capabilities to support process automation and optimization. These capabilities include modeling business processes, automating the processes


��


directly from the model, and monitoring running processes. Available solutions have varying levels of support for managing manual processes as part of the overall process and for using simulation capabilities to optimize processes. Another evolving capabil-ity is the ability to externalize business rules rather than having them hard-coded into the process integration software. That way, businesspeople can directly control their portion of the process and thereby increase business agility. For example, during the busy season, a manager could raise the dollar amount that would trigger an approval for an order.

Unlike BPEL, which is gaining mind share as an underlying execution language, no widely supported standards currently exist for process modeling. One candidate, Busi-ness Process Modeling Notation (BPMN), is working its way through the standards process within the Object Management Group (OMG). However, few vendors currently support BPMN. Most have their own modeling notation with BPEL support for process execution.

As mentioned, a number of BPM solutions also include simulation capabilities to facili-tate process analysis and optimization. Process designers can perform what-if analysis to test and compare variations of the process for better productivity and efficiency. Some tools also allow performance statistics from running processes to be fed back into the simulation engine to analyze and optimize processes dynamically. This makes continuous process improvement possible and is a key to gaining competitive advan-tage.

BPM solutions can also be used to create composite applications in the form of busi-ness processes that span multiple systems. The BPM tool can model and implement the flow of control across services and applications. BPM products on the market today generally provide additional capabilities beyond simple process orchestration in the areas of process monitoring, management, and optimization.

WOrkflOW managemenT

Workflow management solutions focus on the efficient coordination of manual tasks. These solutions have been available for many years and predate most BPM solutions.


�0


However, because end-to-end business processes frequently include automated as well as manual processes, many integration vendors now offer BPM and workflow manage-ment in a single platform. Vendors that focus exclusively on workflow management are rapidly decreasing in number.

BPM solutions vary in the level of support they provide for manual processes. Robust workflow management involves process modeling as well as the design and develop-ment of user interfaces to manage task lists, accept or change assignments, balance workloads, and account for changes in resource availability due to vacations, illness, etc. Manual workflows often involve approvals, which need to be logged and audited and require the workflow solution to understand the organizational hierarchy and ap-proval chain. In addition, different interfaces should be available for managers and the employees receiving the tasks.

COllaBOraTIOn plaTfOrms

Also known as groupware, collaboration software facilitates interactions among peo-ple. In contrast with workflow, where the interactions follow established processes with predefined paths and steps, the interactions in a collaboration scenario are generally free-form and ad hoc. Although collaboration is not usually considered a mainstream component of business integration, it is relevant to organizations seeking to imple-ment integration solutions as part of a strategic initiative to optimize business process-es. This is especially true as project teams increasingly become physically distributed. Collaboration is also relevant to B2B, where more specialized solutions enable partners to work together on product design or development, and other joint initiatives. Col-laboration platforms include repositories for managing team artifacts, such as docu-ments and design specifications; version control and security capabilities; and features for managing email threads and discussion groups.

Business Optimization Services

Business optimization services enable organizations to align business strategies with coordinated actions. The goal of BPM is to streamline processes, but optimizing indi-


�1


vidual processes does not necessarily lead to optimization at an organizational level. The goal of business optimization is to optimize at the organizational level and to align all parts of the organization to enhance business performance.

Business optimization services include technologies that enable business performance management. As a discipline, business performance management includes methodol-ogy—such as Six Sigma, Balanced Scorecard, and Lean—best practices, and technol-ogy infrastructure.

Several developments have emerged on the technology front to support business performance management. The most significant of these are dashboards for execu-tives, line-of-business managers, and IT operations staff that provide real-time visibility into the organization’s operations and associated KPIs. This capability is provided by an emerging technology called business activity monitoring.

The well-known analyst firm Gartner coined the term business activity monitoring in 2002 and defines BAM as “the concept of providing real-time access to critical busi-ness performance indicators, along with the supporting information to improve the speed and effectiveness of business operations.” Advances in integration technology have allowed organizations to instrument applications to deliver real-time informa-tion associated with significant business events, such as the receipt of an order. BAM is the technology that monitors the events, correlates them to identify trends—such as a sudden, unexpected drop in order volumes—provides real-time analytical capabili-ties and management dashboards, and alerts users when exceptions occur. Some BAM solutions include capabilities to predict when exceptions are imminent by comparing current operating metrics with historical norms.

BAM solutions are emerging from BPM, integration, database, business intelligence, and systems management vendors. The market for BAM is a quickly evolving sector, and one that is not yet fully defined. The key feature requirements for business perfor-mance optimization include dashboards, statistical analysis of user-defined KPIs, auto-mated and dynamic performance thresholds, and advanced correlation and predictive monitoring to help managers identify developing concerns and react before they be-come full-blown crises.


��


Ultimately, business performance optimization must encompass all levels of the orga-nization, both on the business side and within IT. For example, an emerging category of solutions from software management vendors is business transaction management. This technology enables systems administrators to view the status of end-to-end trans-actions as they cross systems on a business process level so they can better understand the context of the transaction.

Compliance and Industry Solutions

The previous categories all represent integration infrastructure services. They are the technologies to use when creating business solutions. The Compliance and Industry Solutions section of the Roadmap encompasses business applications that utilize the integration infrastructure technologies to provide specific business applications.

The difference between these solutions and traditional business applications lies in their process-centric nature and in the fact that the processes typically span multiple underlying systems, organizational groups, and other entities. In other words, these are inherently integrated applications. Compliance and industry solutions require an inte-grated infrastructure—including process monitoring, tracking, and auditing—because none of the individual participating systems has the total end-to-end view.

Compliance is an example of a business requirement that involves integration across a diverse set of areas and has a strong monitoring and auditing requirement. Many inte-gration vendors and systems integrators now offer Sarbanes-Oxley, Basel II, and Health Insurance Portability and Accountability Act (HIPAA) solutions that make it faster and easier for organizations to implement solutions to comply with government regula-tions. Similarly, integration vendors and consultants are offering solutions for specific industries—including healthcare, manufacturing, finance, and retail—as well as solu-tions for supply chain management and customer relationship management.

Organizations seeking to implement new business solutions that require integration will focus on vendors in this category.


��


Management and Security

Every level of the integration infrastructure requires management and security. Emerg-ing management offerings include third-party solutions for application integration platform management, Web services management, and end-to-end transaction man-agement at a business process level.

With the growing adoption of SOA, the need for Web services management has be-come increasingly important. Web services management frameworks help users deal with issues such as registering Web services across the enterprise, monitoring those Web services in operation, and determining problems, as well as related areas such as governance and security. Without management, the ad hoc creation of Web services across the enterprise can quickly result in chaos, with no one, for example, having a complete understanding of what is available, who is using what service, and which services are secure.

Security is another extremely important aspect of integration, and integrated security solutions are beginning to emerge. Solutions that include digital certificates and en-cryption are especially important for B2B security. Where standards are lacking, vendor solutions are filling in the holes, especially in the area of Web services.

Architecture and Development

The ebizQ Integration Roadmap helps explain the types of integration technologies available. Some vendors offer only one type of technology, while others offer platforms that incorporate many aspects of the Roadmap in their solutions.

Whether an organization is partnering with one vendor or several, taking a strategic approach to architecture and development is a key element for success in implement-ing the technologies represented in the Roadmap. Organizations that take a tactical approach to developing integration solutions risk suffering the same problems expe-rienced with developing distributed applications: multiple redundant technologies used for different implementations, resulting in increased maintenance costs and a more fragile architecture. Organizations that take a strategic approach to developing


��


the architecture start with an overall plan and then implement it on a project-by-proj-ect basis. This prevents the purchase of redundant technologies for different projects, avoids overlaps in integration technologies, reduces training and maintenance costs, leverages IT investments, and increases business agility. Rather than requiring individ-ual project teams to research and choose their own integration technologies, they can simply use what is already in place and focus instead on adding business functionality and value.

A well-defined and integrated technical architecture also increases flexibility in the de-velopment phase. If the business services conform to standard interfaces, they can be written in a variety of languages, enabling organizations to leverage skill sets within the organization. For example, services written in Java can be part of the same compos-ite application as services written in .NET.

Organizations that wish to take a strategic approach to developing an integrated in-frastructure should establish a centralized group. Some organizations may place this group within their Enterprise Architecture Group, if they have one. However, since inte-gration requires specialized skills, many organizations are establishing an Integration Competency Center (ICC). Chapter 6 focuses on best practices for creating an ICC.

Conclusion

Integration is a journey consisting of many tactical implementations. Organizations that take a strategic approach to building an integration architecture will be better able to avoid integration mistakes, control redundancy, lower the cost of ownership, increase ROI, and ultimately achieve greater business agility. A key element of this is understanding the role of different integration technologies and their context within the overall integration infrastructure. The ebizQ Integration Roadmap can guide you in this process.


��

Executive Overview

Service Oriented Architecture (SOA) is transforming IT, so it is important to under-

stand the relationship between integration and SOA. This is not a straightforward

task, however, because definitions and interpretations of SOA abound.

From a software engineering perspective, SOA is simply an approach to software de-

sign that involves assembling systems from reusable components (services) that may

originate from different sources and underlying technology environments. At this lev-

el, SOA is often associated with Web services. However, Web services address only the

low-level interactions between services; they do not address the grander architectural

and design philosophies of SOA. In other words, SOA is about much more than just

Web services.

From a broader IT perspective, SOA suggests a fundamental shift in how an organiza-

tion implements business systems—with attendant changes in technology, method-

ology, and organizational structure. An even broader interpretation is that SOA marks

the end of monolithic enterprise applications and the beginning of more flexible and

adaptable business process-centric applications.

While definitions of SOA vary, there is unanimous agreement on SOA’s value and prom-

ise. The past decade has seen unprecedented leaps in computing. More and more as-

pects of a business use computers. Additionally, the Internet has extended system inter-

faces to include connections with trading partners. Not surprisingly, this has resulted in

an unruly explosion of stovepipe applications, databases, and electronic connections.

The challenge for IT today is to create and support an infrastructure that can harness

these diverse assets and deliver on new business requirements more effectively. SOA

provides the basis for this infrastructure and offers the promise of tangible business

benefits. SOA also makes it possible for organizations to leverage IT assets so they can

rapidly deploy new business solutions as needed.

Integration and SOA5


��

Chapter 5Integration and SOA

With its promise of transforming the way systems are developed, SOA will benefit how

organizations approach integration. At a basic level, the growing adoption of Web ser-

vices will improve system-to-system interoperability—making it simpler to connect

systems—while the componentization of business services will make it easier to cre-

ate new business solutions from existing applications and retrieve information as the

business demands. At the same time, integration technologies play an important role

in the SOA portfolio, addressing needs such as service orchestration, service- and pro-

cess-level monitoring, service management, and the ability to make legacy systems

part of the SOA environment. Clearly, integration and SOA are highly complementary,

with SOA making it easier to integrate systems and business processes, and integration

technologies providing the essential technical infrastructure services required for SOA

implementations.

Unlike previous IT revolutions that were accompanied by massive change, SOA enables

an evolutionary approach and the opportunity for organizations to move forward in-

crementally. The key to success is to plan strategically at an enterprise level and then

implement tactically, with the appropriate best practices and governance policies and

procedures in place. This strategy delivers benefits along the way, making it easier to

justify SOA investments.

This chapter explores the benefits, design concepts, technologies, and best practices of

SOA, and shows how they relate to integration.


��


Case Study 5-1 Company: WRA Electronics

Company Description

WRA Electronics is a pseudonym for a Fortune 500 global provider of electronic components and computer products. The company serves as a supply channel partner for more than 500 suppliers and 150,000 original equipment manufacturers, contract manufacturers, and com-mercial customers. Headquartered in New York, WRA employs more than 10,000 people in more than 200 locations around the world.

Business Challenges

As a player in the global electronic equipment market, WRA is required to comply with RoHS, a European industry directive restricting the use of six hazardous substances in electrical and electronic equipment. As part of the directive, WRA needed to make all relevant information about its products accessible to its customers for quoting, ordering, and purchasing decisions. However, different business processes (e.g., quoting and ordering) employed different sub-systems running on different platforms and diverse technologies, and the global components data was located in an Oracle database.


WRA adopted an SOA strategy to address its requirements. Using webMethods as its SOA platform, WRA was able to link its disparate systems using Web services and to provide a service-oriented interface for consumer applications. The webMethods solution also provides connectivity to WRA’s mainframe-based systems that currently cannot interoperate with Web services. The RoHS Web service provides data from the Oracle database and plugs into the ex-isting business processes for quoting and ordering. Any differences between the platforms are bridged using the Web service interface, and the webMethods platform performs all necessary mappings and transformations.


WRA is now able to comply with the RoHS mandate. The new system provides global en-terprise visibility into the company’s products and parts along with RoHS information. Each month, more than 500,000 transactions pass through the company’s internal Web services integration environment, and the RoHS Web service is utilized more than 80,000 times.

In addition to complying with the directive, WRA has achieved significant cost savings and efficiency improvements with its SOA approach. The system has enabled the automation of numerous process steps, saving time and errors. Company sales personnel have also been able to help customers make more informed purchasing decisions, increasing customer satisfaction. By providing its customers with quick and easy access to RoHS-related information, WRA has gained a competitive edge in the market.


��


Why SOA?

SOA provides many benefits to both IT and the orga-nization at large. The most im-

portant benefit—and the one most often cited in relation to SOA—is reusability. SOA

allows organizations to do more with the resources at their disposal. It promotes reus-

ability by enabling services to be shared by different applications running on different

platforms. Reusability creates a competitive advantage by providing greater flexibility

in the way computer systems can be used to support the business.

Figure 5-1. Flexibility of SOA through reuse

For example, many organizations have stovepipe applications that address specific ar-

eas of business functionality, such as order processing, invoice handling, or customer

relationship management (CRM). Each application has its own customer data and its

own view of the customer. And each has local mechanisms for retrieving information,

such as customer details, that were developed in the environment and technology

specific to the application platform. In an SOA, these duplicated routines can be re-


��


placed by a common reusable service (for example, FindCustomer) that delivers the

same functionality. Because there is just one copy of the FindCustomer service, chang-

es need only be made in one place. This reduces application maintenance costs and

enables new changes to be accommodated more quickly and easily.

Another benefit of SOA is increased agility when addressing business opportunities or

changes in business requirements. By reusing common steps and developing only the

new steps that are required, organizations can more easily modify or enhance business

processes and applications when the need arises. For example, a company that offers

home loans may have a loan approval process consisting of numerous SOA services.

If the company now wants to offer car loans, it can reuse many components from the

existing implementation, such as checking current account details or market interest

rates. This increased agility translates directly into increased competitiveness. Thus, or-

ganizations that master SOA will gain an advantage over those that do not.

A third benefit of SOA is the reduced complexity from a more orderly enterprise archi-

tecture and development approach. Modular services enable a less-complex, divide-

and-conquer approach to software development. This enables SOA to accelerate the

deployment of new application functionality, decreases the testing effort by making it

easier to test individual components independently, and lowers implementation risks

through the reuse of functionality that is already quality-tested and proven.

Another benefit of SOA is increased IT adaptability. Packaged application implementa-

tions, custom-built solutions, and system changes resulting from mergers and acquisi-

tions can be integrated more quickly and easily. SOA’s inherent platform-independent

approach provides organizations with more flexibility to use the software and hard-

ware of their choice. This allows them to engage in a multisource strategy and reduces

the threat of vendor lock-in. For solutions developed in-house, it enables programmers

to use their existing technical skill sets.


�0


Figure 5-2. Adding new functionality easily

Best of all, these benefits can be realized quickly. SOA is ideally suited for incremental deployment, with investments made step-by-step for individual projects. Returns can also be realized incrementally. This eliminates the need for costly and risky “big-bang” software implementations, reduces investment risk, and delivers value more quickly. Thus, SOA becomes a far more palatable investment for organizational decision mak-ers.

SOA Explained

Although the term Service Oriented Architecture first emerged in the 1990s, the spirit of SOA existed long before that. The underlying principles of SOA—componentiza-tion, encapsulation, separation of interface from implementation, loose coupling, and so on—have been promoted since computing’s infancy. These design principles were used in the days of centralized computing, when the focus was on IT resource optimi-zation.


�1


Then the landscape changed. On the technology side, the arrival of distributed com-puting offered great productivity and price performance, and the emergence of the In-ternet promised ubiquitous connectivity. On the business side, IT was increasingly seen as something that could be used for business advantage rather than just back-office operations. But these changes introduced considerable challenges in linking together different technologies and platforms to utilize the increasingly diverse IT infrastructure to meet new business needs. The advent of object- oriented computing paved the way for SOA by introducing the concept of having discrete program components with standard inputs and outputs that could be used to assemble applications.

So what exactly is Service Oriented Architecture? From a software engineering stand-point, SOA is an IT architecture based on the delivery of reusable, well-defined busi-ness services underpinned by IT components in such a way that the providers and the consumers of the business services are loosely coupled and have no knowledge of one another’s technologies, platforms, or locations.

An explanation of the term “business services” is helpful here. With SOA, IT compo-nents are formed into services that are relevant to a particular business operation. For example, the service for FindCustomer might involve code that locates the customer number by searching for the customer’s name in one system, then more code that uses the customer number to query the customer’s service subscriptions in another application, and yet more code to retrieve the organization’s interaction history with the customer from a third database.

A service consists of a well-defined service interface and the service implementation. The service interface describes how to call the service. It specifies, among other things, the location of the service and the information required to make a request to the ser-vice and get a response. This is where Web services provide an ideal fit for SOA. The use of Web services ensures a standard protocol for communication between the service consumer and provider and a standard way to define a service’s interface. It is basically an electronic description of how to call the service from other programs.

The service implementation is the actual code that fulfills the functionality of the ser-vice. It is the logic that resides on computers somewhere on the network and executes


��


when the service is called, subject to appropriate security constraints. The service im-plementation is inherently platform-dependent. However, SOA is not concerned with how services are implemented. Architecturally, it does not matter, for example, whether the FindCustomer service is written in Java, C#, or COBOL. The only thing that matters is that the service fulfills the behavior specified by its interface.

Another important term used when defining SOA is “loose coupling.” Services that are loosely coupled can be invoked by any application on any platform in any location. All the calling application needs to know is how to invoke the service, and any inputs and outputs required. This enables the interoperation of different applications and services written in different languages with different development tools and platforms. Loose coupling also means that one service can be replaced by another—provided it has the same signature (inputs and outputs)—without impacting any of the other connected services or requiring them to be modified. That means incremental changes to a sys-tem can be made without having to redevelop the entire application. The dynamic nature of SOA is another important distinction from more traditional application de-velopment approaches.

A final consideration when thinking about SOA is “granularity.” Granularity refers to the level of functionality provided by a service. A fine-grained service (such as Veri-fyZipCode) performs a very specific and discrete function, while a course-grained ser-vice (such as ProcessInvoice) incorporates more functionality and typically maps more closely to the notion of a business service. This distinction sets SOA apart from previous computing paradigms by providing a common language for IT and the line of busi-ness to discuss computing requirements. Rather than talking about low-level bits of programming code, SOA allows application requirements to be expressed in terms of modular, reusable business services. At a technical level, SOA enables services to be made up of other services, which makes it possible to assemble course-grained servic-es that map to useful business functions. An important goal when designing services is to encapsulate business functionality at an appropriate level of granularity so it can be reused across different solutions.


��


The Role of Web Services

As mentioned, SOA is often discussed in conjunction with Web services, but the two

are not synonymous. In theory, SOA does not require Web services, and simply imple-

menting Web services does not result in an SOA. However, Web services are the first

standard for service-oriented computing to gain the near-universal support of soft-

ware vendors, end-user organizations, and standards organizations. By standardizing

essential elements of SOA, Web services remove the risk of having to bet on a particular

technology (as was the case with CORBA in the days of distributed computing). As a

result, Web services and their related standards are driving the widespread adoption

of SOA.

The Web services standards define XML-based protocols and interface descriptions

that enable services to interact. Basic Web services standards include Simple Object

Access Protocol (SOAP), Web Services Description Language (WSDL), and Universal De-

scription, Discovery, and Integration (UDDI).

XML, or Extensible Markup Language, is a World Wide Web Consortium (W3C)-recom-

mended standard for defining different types of data. XML provides a text-based way

to describe and apply a tree-based structure to information. It has several advantages

over previous approaches to describing data: It is readable by humans and machines, it

is platform-independent, and it has the flexibility to handle arbitrary types of data and

data structures. Propelled by the adoption of the Internet and open standards, XML has

emerged as the critical standard for data definition and document markup.

SOAP is a lightweight protocol intended for exchanging XML-based messages in a

decentralized, distributed environment, normally using HyperText Transfer Protocol

(HTTP), the protocol used between Web servers and browsers on the Web. HTTP was

chosen as the primary application-layer protocol for SOAP because it works well with

today’s Internet infrastructure and is firewall friendly—unlike other distributed com-

puting protocols, which are generally filtered out by firewalls.


��


A SOAP message consists of a one-way transmission from a SOAP sender to a SOAP

receiver. Applications can use SOAP to create more complex interaction patterns, in-

cluding request/response and multiple, back-and-forth (conversational) exchanges.

The most common pattern is the remote procedure call (RPC) or request/reply pat-

tern, where one program (the client) sends a request message to another program (the

server), and the server immediately sends a response message to the client.

WSDL is an XML-based format for describing Web services. WSDL defines the public

interface to a Web service, which describes how to communicate with the Web service

as well as the operations and messages supported by the Web service. WSDL is typi-

cally used in combination with SOAP. A client program reads a Web service’s WSDL to

determine what functions are available on the server, and then uses SOAP to actually

call one of the functions listed in the WSDL.

UDDI is an XML-based standard for a Web services registry (i.e., directory) that allows

service providers to publish information about their services and enables potential

service consumers to browse listings of available services (using SOAP messages) to

identify the ones most suited to their application requirements. As such, UDDI is often

described as a Yellow Pages for Web services. Originally intended as a public resource,

UDDI is now more commonly used within organizations as a private registry of their

internal Web services.

Additional Web services standards are being developed in areas such as security, reli-

able message delivery, and transactions. Standards are also being created to define

the business process flow across services (for example, BPEL, which was discussed in

Chapter 4). Unfortunately, as history has taught us, standards alone do not guarantee

interoperability. Vendors may implement subsets of the standards for expediency, or

supersets of the standards for enhanced functionality, but these subsets and supersets

may actually limit interoperability.


��


Figure 5-3. How Web services standards interact

To guard against interoperability issues and the potential slowdown in Web services

adoption that might result, the vendor community has formed the independent Web

Services Interoperability Organization (WS-I). WS-I’s charter is to promote the interop-

erability of Web services implementations from different vendors. WS-I creates profiles,

which are implementation guidelines for interoperability, and testing tools to verify

conformance with a profile and WS-I guidelines. Currently available profiles include the

Basic Profile, Attachments Profile, and Simple SOAP Binding Profile. In addition, work is

under way on a Basic Security Profile. WS-I profile conformance is an important con-

sideration for organizations that want to ensure interoperability of their Web services

across different vendor platforms.


��


Case Study 5-2 Company: U.S. Power & Light

Company Description

U.S. Power & Light is a pseudonym for one of the country’s largest power providers. The company manages a distribution grid that spans 11 states and provides electricity to 5 million customers.

Business Challenges

Several years ago, the Federal Energy Regulatory Commission mandated that U.S. Power & Light become a member of a regional transmission organization (RTO). RTOs are indepen-dent, government-regulated organizations focused on energy transmission and distribution for regions of the country. U.S. Power & Light needed to meet the RTO’s stringent technical requirements in 10 months, or it risked federal fines of $1 million a day. These requirements included providing real-time and batch information on capacity, number of units online, power produced per hour, emissions, and outages directly from the company’s power plants. At the same time, U.S. Power & Light needed to improve its operational efficiency to meet Sarbanes-Oxley requirements. Redundancies in the existing system required employees to enter the same information into 15 different systems for some transactions.


More than 100 new interfaces were required for the types of data and forms that U.S. Power & Light would be exchanging with the RTO. The company chose webMethods as the anchor of an SOA strategy that enables it to publish data in one place to support the RTO require-ment. With the webMethods platform, all subscribing systems can now instantly retrieve the information they require. The solution is based on XML schemas, which are translated from the source system into U.S. Power & Light’s internal canonical forms. This enables the data to be integrated into multiple systems without point-to-point data transformations. With the new system, U.S. Power & Light sends its RTO hourly schedules of where it will be delivering power and where it will be pulling power off.


The SOA-based webMethods platform has enabled U.S. Power & Light to be development platform-agnostic for application interfaces. Using Web services, the company can easily con-nect disparate systems and reuse code when automating new processes. U.S. Power & Light met its 10-month schedule and avoided $1 million per day in potential fines. In addition, it can now meet the Sarbanes-Oxley requirements. With all the integration points in its systems con-nected through a common platform, U.S. Power & Light can easily monitor and document the system and processes, errors, and other key areas as required for compliance. SOA and Web services were two key ingredients for this success.


��


Integration’s Role in SOA

Although SOA is commonly associated with the creation of new systems, SOA and in-tegration are also closely linked. A key driver of SOA is interoperability, which of course is at the core of integration. In driving a greater degree of system-to-system interoper-ability through the adoption of Web services standards, SOA will ease the challenge of integration. For reasons explained below, however, it will not replace the need for higher-level integration capabilities. Still, organizations can increase flexibility and re-use by taking an SOA approach to implementing integration projects. In essence, SOA makes integration easier, and integration technologies enable SOA to achieve greater productivity and utility.

So how does integration serve an essential role in SOA?

For one thing, as stated earlier, Web services on their own do not provide the full spec-trum of functionality required in an enterprise SOA. Beyond the interoperability en-abled by Web services, additional capabilities are required to assemble, orchestrate, and coordinate Web services into more solutions in real-world business scenarios.

For example, it will be a very long time, if ever, before all enterprise data is accessible in XML format, the data format for Web services. Different applications—built with dif-ferent business requirements for different purposes—define business entities (such as customers, products, or financial securities) their own way. Consequently, translation and transformation of data into the native formats of each application is an essential component of integration today, and will remain an essential component of SOA. Dif-ferent data representations—in terms of syntax (the format of the data) and semantics (the meaning of the data)—will exist as long as more than one system is involved. Since so much data is represented in non-XML formats, XML-based translation and transfor-mation technologies such as XSLT do not suffice. Therefore, integration technologies—with their robust support for complex legacy data formats—are needed to provide SOA with a wider range of translation and transformation capabilities.

Additionally, forward-migration mechanisms are needed to bring the legacy environ-ment—that is, the non-XML, non-Web-services-based systems that will continue to


��


constitute the majority of enterprise systems for a long time—into the SOA fold, since starting with a clean slate is not a viable option. This forward migration is often ex-pressed in terms of putting Web services wrappers around legacy interfaces to make them available to the SOA world.

Though jumping from an existing legacy interface to one based on Web services may be technically feasible, it may not be desirable from a financial or risk-management perspective. Creating Web services interfaces for existing systems requires consider-able development and investment. If the service will be reused often, the investment will pay off over time. For some implementations, however, getting a new solution up and running quickly is the prime requirement. When connecting to enterprise resource planning, packaged, and legacy applications, it may be faster and easier to use existing application adapters provided by an integration vendor than to recreate access via Web services interfaces. Using application adapters for tactical solutions while taking an evolutionary approach to creating Web services from existing applications will al-low an organization to realize an early return on investment (ROI) from SOA imple-mentations. Some integration vendors enable a more seamless transition by making operations exposed through an adapter automatically available as Web services.

Finally, Web services do not provide the requisite infrastructure services for organi-zations focused on using SOA to support a strategic initiative to optimize business processes. Process optimization requires managing the end-to-end business process, which crosses application and organizational boundaries. This is what business pro-cess management (BPM) systems do. Web services orchestration defines the flow of control for a composite application, but BPM goes further with real-time monitoring management, and process optimization through process automation, human work-flow management, and process simulation.

The relationship between SOA and integration can be summed up as follows:

SOA and integration are highly complementary. On the one hand, SOA simpli-fies integration by driving interoperability between systems via Web services, and by componentizing application functionality into business services, mak-ing systems more accessible to integration. On the other hand, integration technologies and solutions provide a migration path to SOA from the existing installed base of pre-SOA systems.

4


��


Interoperability and integration are not equivalent. Web services promote cross-platform interoperability, but true business integration demands higher-level capabilities such as process orchestration, process monitoring, data rec-onciliation, and exception management. These capabilities are necessary ele-ments of SOA-based business solutions, but they are not features provided by Web services. Integration technologies and solutions exist to fill these needs.

The disciplines behind integration and SOA-based development are similar, and best practices developed for integration can be applied to SOA. Organiza-tions that take a strategic approach to integration understand that success lies in appropriate methodologies, governance on the use of organizational IT as-sets, and dedicated organizational structures and roles. These same principles apply in practice to SOA. For example, the expertise and processes developed to manage integration interfaces—to ensure that they are well specified, meet required service levels, and so on—can be applied directly to a registry of busi-ness services. In short, the expertise and competencies developed for integra-tion give organizations a head start with SOA.

Enterprise-Class SOA

As stated earlier, simply making Web services available on the network does not create an SOA. For an SOA to be effective in the real world, numerous infrastructure capabili-ties are needed—such as data translation and transformation, process orchestration, and the other integration capabilities described in the previous chapter—as well as capabilities for security, versioning, auditing, management, monitoring, and ensuring predictable quality of service. The latter capabilities are especially important for en-abling SOA to address mission-critical business requirements.

It is useful therefore to think of an enterprise-class SOA as one that not only incorpo-rates the business services underlying the applications delivered via the SOA, but also provides important infrastructure capabilities for delivering the reliability, scalability, and manageability that global organizations expect. A coherent SOA infrastructure will also help manage the complexity of the environment, without which sustainable reus-ability would not be possible.

4

4


�0


Case Study 5-3 Company: State Bank

Company Description

State Bank is a pseudonym for a European bank based in Germany. With a focus on midsize enterprises, the bank offers its domestic and international customers the financing services of an internationally oriented credit institution and the investments and services of an invest-ment bank. State Bank has more than 1,700 employees and over €70 billion in revenues.

Business Challenges

State Bank was introducing SAP’s Bank Analyzer, which includes a financial database (FDB). The FDB was to be the bank’s new core database, and all enterprise applications needed to be connected to it. This requirement made State Bank’s previous interface model obsolete be-cause it was time-consuming and led to a single point of failure.


State Bank decided that adopting an SOA would shorten the development time of the in-terfaces, making them reusable and easier to maintain while eliminating the single point of failure. The bank implemented webMethods Fabric as the SOA-based integration layer for connecting its legacy systems and databases to the FDB, and used XML to map all the data entering and leaving the integration layer. As a result, all of the bank’s systems can now com-municate with the FDB for the SAP solution.


The SOA-based integration solution has replaced State Bank’s obsolete interface architecture with a flexible, reusable system. After only a three-month implementation phase, the new platform is already processing 170,000 transactions each month. As a result of its approach, State Bank can more easily synchronize customer information with the customer master data, obtain foreign exchange prices in real-time, and update new securities holdings after every transaction.


�1


sOa InfrasTruCTure

Deploying an SOA infrastructure as part of the overall SOA implementation strategy is vital if organizations are to realize the benefits promised by SOA. Although a tactical approach might be possible in the early stages of an SOA implementation, long-term ROI requires a proactive approach to creating the SOA infrastructure.

An enterprise-class SOA infrastructure:

Promotes architectural consistency by ensuring that Web services are de-ployed in a manner consistent with a service-oriented model

Facilitates the assembly of large-scale, dynamic systems from Web services

Provides a central point for managing and monitoring deployed services

Delivers the performance and robustness required for business-critical sys-tems through features such as load balancing and automatic failover between services

Provides mechanisms whereby services can easily locate one another, regard-less of where they are deployed in the network, and promotes loose coupling of business-oriented services through location transparency and dynamic binding

Provides metadata management and registry services to support the reusabil-ity of services

Provides a framework for creating and deploying infrastructure services—ser-vices that function between business-level services, such as transformation and filtering

Allows services to be provisioned dynamically as business needs fluctuate

Provides security across services, addressing needs such as client authoriza-tion

Offers various levels of logging—for example, Web services requests and re-sponses, session-related information, and security events—to support trend analysis, exception handling, and problem diagnosis

4

4

4

4

4

4

4

4

4

4

4


��


An important requirement of an enterprise-class SOA infrastructure is that it exist in-dependent of the business services and applications it supports. As the figure on the next page illustrates, different Web services containers may have different capabilities (including no native Web services capabilities at all). In addition, even when different containers have similar capabilities (for example, support for a specific WS-* standard), the actual implementations may vary or be incompatible because of version differ-ences.

As a result, the only way to achieve the appropriate consistency across different service containers for functions such as security, transactions, service monitoring, and quality of service is to externalize the SOA infrastructure from the service providers. Conceptu-ally, this means that instead of consumers calling Web services providers directly, calls are made through an interception layer that provides a consistent set of infrastructure capabilities across service providers. In addition to enabling consistency, this layer en-ables the SOA infrastructure to do things like load-balance across different services and provide Web services capabilities to systems that are not natively built on Web services.

Figure 5-4. Enterprise SOA infrastructure


��


As explained, the SOA infrastructure should be non-intrusive to the business services themselves. It should also be capable of operating natively within supported platforms, such as an application server, and as an intermediary for nonsupported platforms, such as a packaged application or a service hosted by a third party). With this flexibility, an SOA infrastructure can bring all the services in an organization’s environment under a single management scope.

sOa gOvernanCe

Providing the capabilities of an enterprise-class SOA infrastructure requires a combi-nation of standards, technologies, policies, and procedures. Although the appropriate technologies are important, corresponding policies and procedures are just as crucial to realizing the benefits of SOA. As Web services begin to proliferate in organizations and services are reused in different applications and different areas of the organiza-tion, governance becomes an essential part of the SOA strategy. So does the need to enforce policies and procedures throughout the development and maintenance life cycle of a Web service.

One important area of SOA governance is managing reuse. As experience with object-oriented programming has shown, if ROI is to be realized, the discipline of reuse must be designed into the process. Reuse does not happen by accident. When the prolif-eration of Web services is unmanaged, it leads to duplication, overlap, and an over-all loss of control that erodes the reusability benefits of SOA. Managing reuse means architecting services so they can be reused, monitoring how services are reused, and providing appropriate incentives for reuse. Although these issues are primarily ones of methodology and process, some vendors provide capabilities to help manage reuse. For example, by profiling the performance characteristics of a given Web service (such as average response time), a developer can make an informed decision about whether to use the service for a specific business requirement.

An additional governance issue relates to policy definition, implementation, and en-forcement. Consider the FindCustomer service. A company may need to implement a new business policy that requires customer credit card information to be transmitted


��


in an encrypted format to protect customer confidentiality. The developer of the ser-vice can easily ensure that the service receives and delivers only encrypted informa-tion. The challenge is communicating and enforcing the policy change with individuals who are using the earlier, unencrypted version of that service.

A centralized registry helps support the definition, implementation, and enforcement of service policies. Putting policy rules in a centralized governance repository makes them much easier to enforce across applications and platforms. For example, develop-ment policies can be enforced when a service is registered, and run-time policies can be enforced when a service is invoked. Because of this, the UDDI registry is increasingly viewed as more than just a place to catalog services; it is also a strategic asset in the SOA infrastructure from a governance perspective.

Enforcing security policies across services is another important governance issue. A Web service may be created for use within a single department but then adopted by other users, including users outside the organization. Security must be maintained across users. An ad hoc approach to Web services management provides no mecha-nism for implementing a security policy and enforcing compliance across the organi-zation.

A final issue relates to the operational monitoring and management of Web services from a service-level perspective. A Web service may be used by applications across the enterprise that have different quality-of-service expectations regarding availability, performance, and security. It may be possible to monitor the performance of the ser-vice in isolation, but the more important requirement is whether the service satisfies the service-level expectations of a particular user.

When issues such as these are not addressed at an enterprise level, and tactical Web-services-based solutions begin to proliferate without effective governance, the ben-efits of SOA rapidly decline. As such, the more progressive software vendors offer not only technology solutions but also a broader portfolio of capabilities that includes best practices and methodologies for helping organizations realize their SOA strategies ef-fectively.


��


Best Practices in SOA

By now it should be clear that SOA is an enterprise strategy, not a technology or a tacti-cal solution. To reap the rewards of SOA, organizations must implement an enterprise-level approach. Best practices include the following:

Create an enterprise SOA group. This group is responsible for defining the enterprise SOA architecture and roadmap, and managing the SOA’s incremental implementation by working with project groups across the organization. The enterprise SOA group is similar in function and purpose to the Integration Competency Center (ICC), as de-scribed in Chapter 6, and may evolve as an outgrowth of the ICC or the Enterprise Ar-chitecture Group, if one exists.

Manage reuse. As mentioned earlier, reuse must be managed if any benefits are to be realized. Among other things, managing reuse requires planning and designing for reuse, maintaining appropriate documentation and metadata about reusable assets, having a registry of the services available for reuse, and providing active administra-tion. Policies and incentives for reuse are also important.

Control redundancy. Duplication of technical and business services drastically in-creases maintenance costs and complexity. Redundancy usually results from poor governance and a temptation to take shortcuts. These can be avoided with strong en-terprise controls and best practices.

Develop SOA competency within the organization. Loosely coupled Service Ori-ented Architectures provide maximum business agility. However, it is not enough to provide programmers with an integrated development environment for creating Web services. Programmers need to understand how to design and build systems that fol-low and take advantage of the new paradigm. For example, developers should under-stand how to design services at the right level of granularity to maximize reuse. For long-term success, organizations need to invest in developing a core competency in SOA design and implementation.


��


Conclusion

SOA is the undisputed path for creating an agile IT infrastructure. Although there are many important technology considerations—including the implementation of the in-tegration technologies represented in the ebizQ Integration Roadmap (Chapter 4)—SOA is about more than technology. The broad, successful adoption of SOA requires a proactive, managed approach that has implications for an organization’s development practices, its organizational structures, and its approach to investment in applications and software infrastructure. Taking the proper strategic approach requires an initial upfront investment, but the benefits accrue as the number of deployments grows. Over the long run, a strategic approach offers a much higher ROI than what might be experienced with quick and easy tactical solutions.


��

Executive Overview

A major theme of this book is that integration and Service Oriented Architecture (SOA) are not shrink-wrapped solutions, nor are they simply about technology. If

an organization wants to reap all the rewards that integration has to offer, it needs to follow best practices, avoid common pitfalls, and understand how to structure the or-ganization for success. Risks include ignoring data quality issues, lacking clear require-ments or well-defined project plans, not having a cohesive organizational structure, and lacking processes and procedures for implementing integrated solutions.

Integration skills are specialized, in demand, and in short supply. One of the more im-portant best practices to emerge is the notion of an Integration Competency Center (ICC) within the IT organization. An ICC provides an environment in which critical skills can be honed, knowledge and experience from previous projects can be consolidated and reused, integration processes can be developed and tested, and governance can be applied to reusable integration assets and business services.

Most organizations will spend considerable sums on implementation and consulting services. However, integration needs to become a core competency within the orga-nization so that new solutions can be easily implemented on an ongoing basis. This chapter addresses some common integration pitfalls, explains the importance of tak-ing a strategic approach to integration, examines various models for establishing an ICC in an organization, defines roles and responsibilities, and identifies the critical re-quirements for successful integration projects.

Common Pitfalls

By taking a few precautions, organizations can avoid the pitfalls that jeopardize an integration project’s success.

Best Practices for Integration6


��

Chapter 6Best Practices for Integration

Data quality issues. Integration is pointless if the underlying data quality is poor. As the adage goes: “garbage in, garbage out.” Before integrating systems, organizations should first assess the quality of the original data to determine whether it meets the usage needs elsewhere. Some level of data cleansing in the source systems is often necessary to avoid downstream exceptions that reduce the efficiency of an automated integration process.

Lack of clear requirements. When an integration project is started while an appli-cation is still being developed or customized, the integration requirements become a moving target that is much harder to hit. Before beginning an integration project, organizations should first ensure that the source systems and interface requirements have solidified. With integration, the devil is in the details, meaning that even minor changes to the requirements can entail significant rework of the design and solution.

Poor project planning. Organizations frequently fail to plan for the long lead times of-ten associated with hardware and software procurement or firewall and other security changes. Consequently, dependency-related delays and scheduling issues are com-mon. This pitfall can be avoided by working closely with procurement, security, and operations management to ensure a smooth process. Another aspect of poor planning is failing to coordinate and establish realistic deadlines when integrating with trading partners. Organizations should work closely with their trading partners to establish realistic project timelines, milestones, and testing processes that factor in the compli-cations of dealing with external entities. They should also communicate frequently to ensure that all parties are on track with the schedule and milestones. When kicking off a project, it is a good idea to meet key trading partner contacts in person. This often reduces the communication barriers during the project.

Poor coordination between IT and business stakeholders. Inadequate communi-cation between IT personnel and key stakeholders in the business opens the door for future scope changes. This is a potential issue on all projects, but integration projects are particularly vulnerable because of the number of related parties involved. Organi-zations should identify the key stakeholders and decision makers for all the interested parties, and establish a formal process to obtain sign-off.


��


Lack of development standards and guiding principles. Naming standards, config-uration management standards, and code promotion standards are critical for agility and reuse. Organizations need to establish standards and guiding principles early to ensure a consistent and orderly solution.

Taking a Strategic Approach to Integration

Historically, most integration projects have been undertaken as tactical projects, or as tasks within a larger implementation project. This is quickly changing because of the inefficiency and cost of taking a piecemeal approach to integration. Treating each inte-gration requirement on a project-by-project basis does not afford the opportunity for reuse from one application to the next. It also increases maintenance costs, since there is no opportunity for reuse and no way to gain economies of scale.

A tactical approach also inhibits the development of integration skills in an organi-zation. Although application developers have been doing integration for many years, most of this has been done in a point-to-point, application-to-application manner us-ing custom code, application- specific programming interfaces, or flat-file data trans-fers. As a result, integration skills tend to be specialized and scattered throughout an organization’s IT community rather than centralized in one group. Organizations today need individuals with the skills required by more sophisticated software and integra-tion platforms that offer built-in support for integration-specific requirements such as transactions, auditing, business process modeling, and process automation. However, few organizations have invested in the resources required to support these new tech-nologies.

The increasing complexity of integration requirements is also fueling the push toward a strategic approach. Rather than focusing simply on linking two or more applications together, integration today seeks to automate process flows both within the organiza-tion and across organizations (for example, integration with customers and suppliers).

If organizations want to more efficiently build and maintain the complex integration processes that their businesses demand, they can no longer treat integration as a series


100


of heterogeneous, project-driven tasks. Organi-zations that take a strategic approach

to integration by creating a separate organizational entity to focus on integration is-

sues and promote best practices throughout the organization will benefit through

lower maintenance costs, greater skills development, and increased business agility.

Organizing for Success

When it comes to creating an organizational structure to support integration, there is

no “one-size-fits-all” approach. An integration team can range from a few resources to

several dozen and can vary in the level of services offered—from simply developing

guidelines and best practices to serving as a full-scale shared services provider that

offers hosting and administration for all of the organization’s integration needs. Re-

gardless of the team’s size or the services offered, best practice recommends that an

organization create an Integration Competency Center. The actual composition and

services of the ICC will depend in large part on the IT organization’s infrastructure. That

is, the ICC model can be centralized, distributed, or a hybrid of the two.

In the distributed model, an organization’s IT resources are divided along functional

areas, product lines, business units, or geographical units. IT groups operate fairly au-

tonomously, and there is little coordination among them. In this model, the ICC estab-

lishes best practices that standardize processes (such as development methodologies

and naming conventions) across project teams.

Although the ICC may not be able to enforce policies and standards in a distributed

IT organization, it can provide a set of integration guidelines for the independent IT

organizations. This increases the likelihood that individual integration projects are ar-

chitected properly. The ICC may also make technology recommendations, though ul-

timately the project teams implement the technology and thus retain a fair degree of

autonomy. The benefit of this model is that knowledge is shared and some procedural

reusability is achieved.


101


In a variation of the distributed model, the ICC defines not only the standard proce-dures but also the standard integration technologies that project teams should use. As before, however, the actual implementations are done by the project teams. Because the integration platform is not under the ICC’s control, the ICC needs to offer services such as capacity planning, disaster recovery planning, and deployment assistance to the external integration teams. The ICC basically acts as an internal consulting group to the distributed IT organizations. This scenario provides a greater degree of consistency than the previous one.

In the hybrid model, some of the integration implementation workload shifts to a cen-tralized group of development resources. As before, the ICC establishes the methodol-ogy and software standards. However, in this model it also contributes to the develop-ment of shared integration assets, with resulting benefits to reusability.

In the centralized model, the ICC acts as a full-service provider. Not only does it set the standards, but it also delivers architecture, design, development, and operations support services. All or most of an organization’s application development and IT infra-structure is consolidated into one organizational IT group. A full-service ICC typically performs the following functions:

Running the integration platform as a public utility. A centrally run and administered integration backbone has been likened to a public utility. Just as a telephone can be plugged into a wall jack and receive phone service, devel-opers can plug systems into the integration backbone and tap into the stream of data flowing through the integration platform. In a highly centralized IT or-ganization, the ICC is responsible for building and maintaining the integration backbone.

Offering standard interfaces to organizational systems. The ICC builds and maintains a library of standard interfaces into enterprise systems that applica-tion developers can reuse in their projects.

Providing data center operations for integration projects. The ICC main-tains the shared integration infrastructure and allocates development, test, and production environments for integration projects. It also assists the IT operations staff with production support and monitoring once the integra-tion project is in production.

4

4

4


10�


Providing integration development services. The ICC acts as an internal consulting group to project teams, providing architecture, design, and devel-opment services either directly through its own staff or by contracting with preferred systems integration partners and off-shore development compa-nies.

Maintaining the enterprise information model. In an environment where multiple integration processes publish and subscribe to data, the ICC must develop and maintain a business object model (also referred to as a canonical data model) of all the documents and data to be exchanged through the in-tegration platform. This role is critical to maintain data consistency and a high level of reusability.

A centralized model provides several benefits. Policies, procedures, and standards are

easily set and administered. Additionally, having all the services under one roof and

point of control maximizes the opportunity for reuse and consistency. The downside of

this model, as with any centralized service organization, is the increased effort required

to effectively manage and respond to the project teams’ commitments and priorities.

In the absence of a formal ICC, project teams operate entirely independently, establish-

ing their own development practices and making one-off technology choices. Allow-

ing project teams to innovate without constraints may have some benefits. But this

scenario runs counter to the SOA philosophies of interoperability, governance, and re-

use. As such, it might be applied to a select subset of projects but seldom across-the-

board.

4

4


10�


Case Study 6-1 Company: MK Healthcare Products

Company Description

MK Healthcare Products is a pseudonym for a Fortune 50 manufacturer of healthcare prod-ucts. The company was founded over a century ago and currently has more than 200 operat-ing companies grouped into segments focusing on pharmaceuticals, consumer products, and medical devices and diagnostics. MK Healthcare Products employs approximately 109,000 people in 57 countries and sells products throughout the world.

Business Challenges

MK Healthcare Products is well known for its highly decentralized management structure. A combination of organic growth, acquisitions, and a global market has increased the need for collaboration of business processes, applications, and data across the organization’s multiple operating companies. In the past, the operating companies developed new processes and services individually, without a common platform, common technology, or business processes. Each operating company had its own integration strategies, architecture, tools, and standards. This often resulted in systems that could not integrate easily with one another, and it signifi-cantly limited the organization’s ability to maximize knowledge sharing, reduce redundancies, and improve business processes.


MK Healthcare Products created an Integration Services Group, whose charter was to create a strategy and platform for integrating business processes and applications. It adopted a total business integration methodology based on an enterprise application integration solution to connect the operating companies with one another and with external business partners. The methodology uses standards, processes, a common design approach, templates, and a philoso-phy of reuse and covers the entire integration life cycle, from project initiation to deployment. By moving toward a centralized approach to enterprise integration, MK Healthcare Products was able to standardize on an integration platform, consolidate hardware, share code and components, combine education and training initiatives, and leverage best practices across the operating companies.


The centralized Integration Services Group has made tremendous gains for MK Healthcare Products. To date, more than 120 projects have been implemented using the total business integration methodology and architecture, resulting in a cost avoidance savings of $9.1 mil-lion. The new, integrated systems process 300 million transactions per week across all projects, which equates to $30 billion in transactions annually—approximately half of MK Healthcare Products’ annual revenues.


10�


Integration Roles

Regardless of the model chosen, an integration team should incorporate the following roles and responsibilities:

Business analyst. With integration requirements increasingly being driven by busi-ness process automation, the integration team or ICC should include resources with strong business domain expertise and analysis skills. The primary responsibilities of the business analyst are to understand the business needs, define the related busi-ness processes and integration requirements, and perform the associated data-level analysis.

Application specialist. The application specialist builds the individual components and Web services from which higher-level integration applications and processes can be assembled. Typically, the application specialist develops business-level services that are associated with one or more applications, so he or she needs to have some level of application domain expertise. Services may be created from scratch (for example, writing a new customer lookup service), by recombining existing services into new services, or by wrappering existing functionality with Web services interfaces.

Integration specialist. While the application specialist focuses on developing reus-able business services, the integration specialist is the middleware expert. The inte-gration specialist develops custom adapters and interfaces, reusable infrastructure services to facilitate different integration scenarios, and other utility integration ser-vices to enable the implementation of more complex business process and integration solutions. The integration specialist also handles the detailed technical engineering of the implementation, conducts capacity planning and other system-level tests, and provides some operational support. The integration specialist’s expertise is generally centralized within the ICC. In some organizations, the application and integration spe-cialist roles are filled by the same individuals.

Process-oriented developer. The process-oriented developer bridges the gap be-tween the business analyst and the application and integration specialists, turning


10�


business process models into executable business processes that link together the ser-vices developed by the application and integration specialists. The process-oriented developer needs both a business-level understanding of the business process and the technical expertise to assemble and orchestrate the underlying services into a pro-cess.

Integration architect. The integration architect plays a critical role within the ICC. This individual defines the overall integration strategy and architecture, performs design reviews of integration implementations, and mentors the rest of the IT organization to ensure that the integration software is appropriately leveraged and properly imple-mented and that integration solutions are architected to take best advantage of the integration infrastructure. In addition to driving standard integration practices and methodologies, the integration architect is often responsible for educating the IT com-munity on the merits of the integration platform and evangelizing the adoption of the software throughout the organization.

Operational support specialist. The operational support tasks that the ICC takes on can vary widely, depending on its scope. Therefore, the number and composition of the operational support team will vary accordingly. At a minimum, the ICC should in-clude resources that can provide integration platform deployment guidance to project teams and data center personnel.

Project manager. The staffing of project managers within the ICC also depends on its charter. If the ICC is responsible for providing hands-on consulting to the IT commu-nity, then project managers need to be on board.

Relationship manager. A strong vendor relationship is critical to the successful adop-tion of new technologies. The relationship manager acts as the central contact between the organization and the vendor. He or she keeps the vendor informed of issues that arise within the organization and notifies the organization about the vendor’s current and future product and service offerings.


10�


Responsibilities of the ICC

The individual responsibilities that an ICC takes on will depend on the organization’s stated goals for the ICC as well as the nature of the organization’s IT group. This section describes the major areas of responsibility for most ICCs and lists possible objectives for each area.

develOpmenT sTandards and praCTICes

An important function performed by an ICC is developing standards to facilitate reuse and interoperability. Responsibilities in this area include:

Creating and maintaining integration best practices—for example, standards for documenting an integration, naming standards and conventions, proce-dures for promoting code to production, and instructions for building security into integration processes

Creating interoperability techniques to standardize the integration of hetero-geneous platforms—for example, integrating between the webMethods Fab-ric platform and IBM MQ Series-based applications

Developing architectural patterns for common integration scenarios—for ex-ample, real-time data lookups, flat-file integration processes, batch integration processes, and transactional business processes

Developing and maintaining an organizational business object model to al-low disparate integration processes to interoperate easily

Developing, harvesting, and maintaining reusable integration components

gOvernanCe

Another important function of an ICC, especially in relation to SOA, is in the area of gov-ernance. Governance—essentially, processes for managing and controlling the use of enterprise assets—is essential for ensuring reusability, long-term maintainability, and productivity. Governance responsibilities of the ICC include:

Establishing policies for how an organization’s services are built and main-tained.

4

4

4

4

4

4


10�


Defining and managing SOA domains, which are sets of services related by a common business context. The domain owner brokers the relationship be-tween line-of-business users and the owners of the applications on which the domain services are built.

Managing the organization’s business services repository and the associated metadata, and ensuring that development practices incorporate the appropri-ate use of the repository.

Establishing service level agreements for business services.

COnsulTIng and TraInIng

The ICC can act as an internal consulting group to an organization’s IT community, pro-viding expertise on the use of integration tools and techniques. Services include:

Providing architectural guidance and review for individual projects

Developing and delivering custom training materials

Providing in-house first- and second-tier support for integration tools

Assigning individual developers to project teams to build integration processes

CusTOmer advOCaCy and vendOr managemenT

As mentioned earlier, continued success with a chosen technology is often predicated

on a strong relationship with that technology’s vendor. To facilitate the relationship,

the ICC should act as the main contact between the organization and the vendor. Re-

sponsibilities include:

Working with the vendor to help market the benefits of both the products and the ICC’s services to the organization

Prioritizing the organization’s enhancement and feature requests, and negoti-ating with vendors

4

4

4

4

4

4

4

4

4


10�


Participating in vendor early-release programs and planning for the rollout and migration of new and upgraded projects

TeChnOlOgy leadershIp

The ICC should constantly look for advances in integration technology that can benefit its organization. The rest of the organization should view the ICC as an expert body for advice on implementing and deploying integration technologies. Services in support of technology leadership include:

Writing white papers on emerging technologies and trends

Hosting internal information-sharing sessions

Staying up-to-date on new vendor features and capabilities

Developing prototype applications using new technologies

OperaTIOnal suppOrT

The area of operational support is where the ICC’s responsibilities are most likely to differ from one organization to the next. At one end of the spectrum, the ICC may own and administer the hardware on which the integration platform is deployed. It would, in essence, create the physical infrastructure on which the enterprise integration back-bone is deployed. Project teams would request development and production space on the shared platform and then turn over their integration processes to the ICC to deploy and monitor in production. In this scenario, ICC services include:

Procuring and maintaining the physical integration platform infrastructure

Providing production support for integration processes built by project teams

Allocating development and testing environments for project teams

At the other end of the spectrum, the ICC could simply provide integration platform deployment guidance to the data center team, which would deploy the integration processes. ICC services in this scenario might include:

4

4

4

4

4

4

4

4


10�


Planning capacity for integration platform components

Planning for disaster-recovery scenarios

Developing fail-over and load-balancing environments

Providing production support for integration platform software—for exam-ple, data center personnel would be responsible for hardware and operations system support, integration developers would support the integration logic they have developed, and the ICC team would be responsible for supporting the integration platform software

Creating an ICC

There are two general approaches to forming an Integration Competency Center. An ICC can be created independent of any one integration project. Or it can be established in concert with an organization’s first integration project and then evolved. We have found that it is better to take an evolutionary approach and allow the ICC to develop from the pilot integration project. This increases the opportunities for success, for sev-eral reasons:

The quality of the standards, practices, and reusable frameworks produced by the ICC tends to be higher because they reflect the team’s actual experience working with the integration tools in a project setting.

Members of the IT community are more willing to accept the ICC’s guidance because the staff has real-world experience.

Management is more likely to approve funding for the ICC because it is con-tributing immediately and directly to project deliverables. As a result, the ICC’s return on investment can be achieved quickly.

Once the original project has been completed, the ICC can be transitioned to an inde-pendent body and can continue refining the methods, tools, reusable architectures, and infrastructures that were established for the pilot project.

The following example illustrates how to build a full-service ICC using an evolutionary approach.

4

4

4

4

4

4

4


110


pIlOT prOjeCT

In this phase, the ICC team may consist of only one or two resources working with the pilot project team. The ICC’s aim is to help develop standard architectures, devel-opment and deployment techniques, and the beginning of an enterprise information model. The ICC staff should include a professional services resource from the integra-tion platform vendor to facilitate the knowledge transfer of best practices and archi-tectural techniques to the ICC team.

TransITIOn TO The IndependenT OrganIzaTIOnal unIT

Once the pilot project has been completed, the ICC moves to an independent organi-zation, gaining resources by transitioning some of the pilot project team members into the ICC. As this phase begins, it is important to formalize the ICC’s charter and clearly spell out the services that the team plans to offer the IT community. In this phase, the ICC:

Assesses best practices and standards developed in the pilot project, makes any changes necessary to render them generic, and publishes them for use on future integration projects.

Builds architectural frameworks around common patterns of integration, or-ganizational systems that are widely integrated to, and common technologies (for example, Web services, MQ Series, and CICS mainframe applications). This will allow future integration projects to be more productive as well as ease maintenance and support, since all integration processes will be based on a common framework.

Begins building the reuse library by harvesting integration components from the pilot project and making them more generic, if necessary, for reuse on future projects.

OngOIng suppOrT

As the ICC becomes well established and its standards and practices are deployed, the team continues to provide support to the organization and:

4

4

4


111


Stays current on integration industry trends and practices

Participates in vendor early-release programs to plan for upgrades and to ad-vise project teams on the use of new features

Continues educating the IT community on integration technologies and best practices

Continues building the reuse library by gathering components from project teams as well as assessing the need for common integration processes and developing them from scratch to offer on future projects

Maintains and grows the enterprise information model to ensure that the in-tegration backbone provides a coherent canonical model for all the applica-tions deployed on it

Maintains the business services repository and expands the inventory of ser-vices available for reuse

Critical Success Factors

Integration can be a complex subject because multiple technologies may be applied to different business scenarios. Regardless of the type or scope of the integration proj-ect, the following factors are critical for success:

Link the ICC mission and charter to business objectives. Even when based on sound technology and executed flawlessly, many a project has been seen as a failure because it did not deliver on the business objectives that the organization was striving to meet. Linking mission to business objectives is even more critical when forming an ICC, which is essentially a cost center to the organization. The ICC’s charter and deliverables must be aligned with the organization’s overall objectives. For example, if the orga-nization is focused on cost savings, the ICC should strive to make integration processes more efficient, reduce implementation costs, and so on. If the organization is focused on im-proving customer service, the ICC should concentrate on creating standard integration processes across all organizational systems that store customer data, building out the enterprise information model to define a single view of customer data, and so on.

4

4

4

4

4

4


11�


Identify and communicate integration and ICC successes. For each project, specific key performance indicators—such as reducing cost or implementation time, or im-proving customer service—should be defined. Then the successes should be tracked and communicated. This will build confidence in the ICC and encourage further orga-nizational investment in integration infrastructure projects.

Build a core competency in integration within the organization. ICC resources should be involved regularly in integration projects so they can continue to increase their skills and add to the collective knowledge of the organization. Systems integra-tors and consultants on an integration project must transfer their knowledge to ICC re-sources. If the ICC does not actively focus on these tasks, integration experience could become scattered throughout the organization and be difficult to leverage across proj-ects. Worse, valuable expertise could leave the organization altogether as the systems integration partners complete their assignments and move on. It’s critical to invest in training to continue building skill sets and knowledge.

Start small and build on successes. Integration is inherently complex. An organiza-tion should not begin by trying to boil the ocean. Instead, it should start small and add integration processes incrementally. Taking on too much too quickly can lead to fail-ure. It is better to underpromise and overdeliver. In the early stages, the ICC should be supplemented with experienced professional services resources from the vendor or a systems integrator to satisfy key project team needs.

Define governance policies and procedures. Many benefits of SOA and integration come from reusing existing organizational assets. However, enabling others to utilize existing functionality for new solutions requires proactive governance of the reusable assets. Policies and procedures should be defined, as should service level agreements. This is a critical success factor for managing reuse.

Obtain executive support. Integration projects routinely cross organizational bound-aries. Automating, optimizing, and managing end-to-end business processes or provid-ing unified access to distributed information can mean stepping on toes. Management support is critical for helping to break down any barriers to acceptance that may exist within the organization.


11�


Conclusion

As organizations invest more strategically in integration and SOA, many decisions and obstacles are sure to present themselves. To successfully navigate these obstacles, it is critical to develop and maintain integration competency within the organization. Es-tablishing and evolving an ICC is a proven approach to promoting best practices; mak-ing efficient use of resources and skills; and improving the consistency, reusability, and agility that enable organizations to reap the rewards of integration. Finally, because of the synergy between SOA and integration and the commonality of best practices, organizations that have made a strategic investment in an ICC will find themselves poised for success with SOA.


11�

Gary So

VP, Office of the CTO, webMethods, Inc.

Gary So is Vice President, Office of the Chief Technology

Officer, at webMethods, Inc, where he is responsible for

advancing the company’s status as a recognized industry

thought leader. Gary has over 10 years experience in

the integration field, serving previously as a system

architect in corporate IT and as a director of professional

services at Active Software, Inc., before joining webMethods in 2000 and spear-heading

the development of the company’s integration methodology. Formerly VP of Strategic

Marketing, he was in charge of driving webMethods’ agenda through media and analyst

relations. Gary has a Masters degree in Computer Engineering from the University of

Toronto.

Beth Gold-Bernstein

VP, ebizQ

Beth Gold-Bernstein is a recognized expert in integration

technologies and SOA. She has over 20 years experience

working with financial institutions, retail chains and

manufacturers on planning, designing and implementing

large scale distributed systems and enterprise

architectures. She has developed training programs for

business analysts, architects, and implementers, and speaks nationally and internationally

at conferences and seminars. Beth is the author of “Enterprise Integration: the Essential

Guide to Integrated Solutions,” published in July 2004 by Addison Wesley. Her first book,

“Designing Enterprise Client/Server Systems,” was published by Prentice Hall in August

1997. Beth holds an M.S. in Computer Information Systems from Bentley College.


concepts, technologies, and best practices - an soa, bpm ... · leading soa and bpm technologies...

Documents