a software development methodology for eaihosteddocs.ittoolbox.com/er011405.pdf · a software...
TRANSCRIPT
A Software Development Methodology for EAI
WORLD HEADQUARTERS5295 Hollister
Houston, TX 77040 USAtel: 832.590.6100
toll free: 1.888.225.3139fax: 832.590.6161
www.zettaworks.com
Copyright © 2004 ZettaWorks LLC. All rights reserved. This material is or contains Proprietary Information,
Confidential Information and/or Trade Secrets of ZettaWorks LLC. Disclosure to third parties and or any
person not authorized by ZettaWorks LLC is prohibited. Use may be subject to applicable non-disclosure
agreements. Any distribution or use of this material in whole or in part without the prior written approval of
ZettaWorks LLC is prohibited and will be subject to legal action.
A Software Development Methodology for EAI
Page 2
The Opportunity
EAI has a proven track record of achieving quick ROI. Success stories of amazing ROI for EAI projects
abound. Gartner found, “As much as 30 percent of the costs associated with implementing major
packaged applications will be consumed by integration. Use of an [integration] broker will reduce these
integration costs by one third. During maintenance, when a single change to an application can have a
rippling effect on several dozen interfaces, use of a broker can reduce costs by two-thirds.”
While the application integration cost savings can be significant, the new architecture of application
integration is driven by organizations seeking competitive advantage. The value transformations
achieved though the automation and integration of an organization’s business processes creates razor
sharp competitive advantage. In addition, companies are extending their business processes through
the value chain to customers and suppliers via the Internet.
The Challenge
While many companies are reporting triple-digit ROI on EAI projects, some have experienced less
than stellar returns or outright failures. This dichotomy of benefits realized is not uncommon for new
technology adoption. EAI is evolving with the continuing introduction of standards (such as Web
Services) and new entrants (application server and pure-play Web Services vendors). The technology
risk of EAI is particularly challenging due to the following factors:
• Organization change in necessary since EAI crosses system boundaries
• The architecture is enterprise in scope encompassing dispersed and heterogeneous systems
• The infrastructure is distributed requiring high availability and scalability
• The project life cycle methodology requires changes due to complex system dependencies, EAI
specific design patterns, and the change impact to the infrastructure and users
• Program management is often complex due the project scope, interdependencies and new
technology risks
• Quality assurance is difficult since EAI components are distributed, have many interfaces, require
new testing environments, and message based testing tools
• New competencies must be developed spanning project management, analysis and design,
development and operations.
A Framework for EAI Success
This paper presents a framework for overcoming the challenges associated with enterprise scale
integration projects. Given that it is difficult for an EAI effort to drive the change of an organization’s
Software Development Lifecycle (SDLC), a pragmatic approach is required with flexibility to change
approach and deliverables. To this end, an analogy is made with modern iterative methodologies where
applicable to EAI projects.
Successful EAI projects are not all about technology, defining the business goals and establishing a
working organization structure are the key first steps. We will examine an EAI methodology from the
perspective of achieving business, organizational and technology goals as follows:
• Establish an Integration Competency Center (ICC) staffed with a cross-function technology team
• Develop a formal EAI business case with strong executive sponsorship, objectives in business
terms and a target Return on Investment (ROI)
• Select EAI tools matching business and architecture requirements to product features and
functions
• Develop an Enterprise Architecture Blueprint based on EAI design patterns and best practices
that defines the EAI logical and physical architecture
• Adapting the current architectures and methodology in use to adjust for EAI – a OOA/OOD
approach with specific EAI design patterns
• Managing iterations of the EAI project lifecycle from analysis to operations
Ideally, an EAI Methodology should improve upon traditional methodologies by addressing specific
issues related to EAI, as well as including best practices, standards, design patterns and processes.
Establishing the Integration Competency Center
Establishing the Integration Competency Center (ICC) is the first step in developing an EAI solution.
Gartner says, “Creating a single organization body that brings the diverse skill set together helps create
the focus, momentum, critical mass, processes and standards required to build and execute world-class
application integration strategy”. Ideally, the ICC should be formed early to define the strategic EAI
business case that drives the software selection process. This would improve the software selection
results and produce an EAI roadmap early avoiding a lag time between software purchase and
implementation improving ROI.
Page 3
A Software Development Methodology for EAI
The ICC seeks to solve the cross disciplinary problems associated with EAI – pulling a team together
with differing critical skill sets. The ICC also provides the role of EAI governance, which applies to project
selection and portfolio management as well as technology standards adoption and management. These
standards include architecture, coding standards, design patterns, reusable components and the Common
Information Model (CIM) – canonical data representations.
Key steps to create and manage the ICC include:
• Create a core ICC team representing business and technical stakeholders prior to software
acquisition to drive the strategic business case, project roadmap and software acquisition
• Establish a Program Manager within the ICC, along with governance policies to manage complex
project dependencies
• Define the roles and needed competences, and create the staffing and training plans to establish
competency
• Address IT cross functional team project requirements early, through the ICC team members, such
as establishing EAI quality assurance policies and procedures, and plans for operations support of
infrastructure monitoring and management
• Establish the baseline EAI architecture
• Within the EAI methodology, create a closed loop process for updating architecture, design patterns,
the CIM, etc., with the governance model within the ICC
The EAI methodology becomes the means for the ICC to define and refine processes such as physical and
logical architecture, project initiation and justification, operational service level agreements, etc.
Developing the High-level Business Case
Many companies take a rudimentary approach to EAI project justification by simply calculating the labor
savings of EAI versus point-to-point integration. However, looking only at cost savings misses opportunities
for the dramatic business process improvement EAI can deliver. EAI is an enabling technology with the
following potential benefits:
• Optimization of assets with real-time logistics management
• Increased inventory turns through B2B integration
• Increased sales by reducing stock outs with real-time inventory management
• Accelerated the integration of mergers and acquisitions
A Software Development Methodology for EAI
Page 4
• Improved visibility of financial data by integrating multiple ERP systems
• Achieved process improvements with Business Process Management
• Improved productivity and access to information with the use of integrated portals
• Enhanced customer service by integrating systems to web based applications
To uncover less than obvious EAI benefits, it is necessary to examine business objectives, applications and
system architecture to develop a high-level business case and feasibility study for EAI. The quickest way to
execute this strategy is through a set of facilitated sessions focusing on business objectives and IT’s ability
to deliver systems that keep up with key business drivers. These sessions examine the strategic value of IT
assets looking for stovepipe applications that act as barriers to the ability to achieve the corporate mission.
The goal of these sessions is to develop a portfolio-level understanding of how the existing and planned
systems meet the organizational goals, and how EAI could accelerate the realization of these goals. The
business case deliverables are a feasibility study covering the following topics:
• Definition of key EAI benefits
• Target applications for integration – a potential project roadmap
• IT organization impact
• High level ROI potential
• ‘Go’/’No Go’ recommendation for a formal EAI product evaluation
EAI Product Selection
Once EAI is determined to be feasible, a formal EAI product selection process should be performed.
Many EAI project are doomed at onset by selecting the wrong EAI suite for the enterprise. EAI suites
have proliferated, ranging from low-end data transformation tools to enterprise class product suites,
including components such as messaging-oriented middleware, integration broker, portal, business activity
monitoring, and monitoring and management. There is a dizzying array of features and functions, emerging
technologies and literally hundreds of vendors professing EAI solutions. And, just picking the market leader
can be problematic since the emerging technologies are often offerings from new players.
EAI suites must fit within the target architecture and support the end applications to be integrated in the
most efficient way. A vendor can have a great EAI suite, but if a single adapter lacks necessary functionality
for a critical business application, the project is at great risk of failure. Also, the physical architecture
features, such as platform support, scalability and fault tolerance can also become major implementation
issues. Likewise, vendors have strengths and weaknesses that must be measured against requirements.
A Software Development Methodology for EAI
Page 5
The following outlines a five step EAI selection process (figure 1) to address the challenging task of EAI
software selection:
1. Architecture and Needs Analysis – Joint Application Design (JAD) sessions to determine
differentiating criteria
2. EAI Business Case Development
3. ROI-Based ‘Go’/ ‘No Go’ Decision
4. Request For Proposal Process
a. Complete and Issue RFP
b. Verify Vendor References
c. Refine Vendor Short-List
d. Facilitate Scripted Demos & Proof of Concept Reviews
e. Create Decision Matrix
f. Cost Negotiations & Final Selection
5. Recalculation of Total Costs (Including Implementation) & ROI
The Architecture Blueprint
It is beneficial at this point to consider the impact of existing methodologies in use with respect to the
creation of EAI specific architecture. The Rational Unified Process (RUP) has a more structured approach
(many would argue heavy) to architecture than agile methods such as eXtreme Programming (XP). XP
��������������������
������������������������
������������������������������������������������
����������������
�����������������
����������������
������������������
����������������
�����������
������������������������������
���������������������
�������������
�����������
��������������������
�������������������
����������������
A Software Development Methodology for EAI
Page 6
proposes quick iterations and frequent refactoring (incremental improvements to internal design) as an
alternative to formal architecture. The concept of refactoring applied to EAI when considering the potential
ripple affect across multiple interfaces when changing things like the CIM or standards such as coding,
message naming or queue topics becomes unwieldy.
With EAI it is necessary to understand the big picture of the target enterprise-level integration for
architecture. This leads to the recommendation to create baseline architecture in the RUP style – creating
candidate architecture in the inception phase and completing the baseline in the elaboration phase. This
works particularly well when a reference architecture for the EAI product already exists (as a vendor or
integrator’s best practice) and needs site specific changes and vetting with a pilot project.
This is not to say that architecture is not an iterative process. Architecture should evolve by continuously
adding to the CIM, design patterns, interface design artifacts, and adjusting infrastructure – within the
governance of the ICC. However, it is necessary to have a starting point for standards and a strategy to
fulfill non-functional requirements on the enterprise scale. The RUP architecture centric approach proposes
an iterative architecture, addressing the most architecturally significant use cases first, which works well for
EAI.
Having addressed the concern of EAI architecture process, it is necessary to consider information
organization and actual deliverable content. With respect to organization, Philippe Kruchten, in his
paper appearing in IEEE Software titled “Architecture Blueprints – The “4+1” View Model of Software
Architecture”, presents an architecture model consisting of concurrent views specific to various
stakeholders as follows:
• Logical view for end-user functionality
• Development view for programmers and software management
• Process view for integrators dealing with performance and scalability
• Physical view for systems engineers dealing with topology and communications
The precept of architecture views is particularly interesting regarding the introduction of EAI to an
organization as a new enabling technology. The organization of the architectural deliverables can target
specific stakeholders to decrease the learning curve. To this end, a set of architecture guides (specific to
enterprise architectures, developers and systems administration) address the typical EAI stakeholders (the
end user or logical view would be expressed within a project specific architecture – not at the enterprise
level). Using this scheme, Kruchten’s architecture model views of development, process and physical are
addressed within the architecture guides relative to their specific stakeholders.
A Software Development Methodology for EAI
Page 7
Given the organizational model of architecture, developer and systems administration the architecture
deliverables follow:
Architecture Guide (deliverable relationships in figure 2)
• Physical and logical design
• High-level as-is and to-be interface diagram
• EAI product roles within the organization
• Common application services – e.g. auditing, error handling, security
• Non-functional requirements and strategy to meet requirements – e.g. fault tolerance, capacity
planning, performance
• Infrastructure – network and platform configurations
Developers Guide
• Naming standards
• Product usage guidelines
• Common design patterns
• Guidelines for reuse
• Change management strategy
Systems Administration Guide
• Build documents
• Monitoring and management overview – e.g. logs, consoles, scripts, admin tools
• Environment management – change control and migration
• Run time component inventory
• Directory structure and security
The ideal scenario for creating the EAI architecture is through a series of JAD sessions with the target
stakeholders and an EAI vendor or integrator’s Subject Matter Expert (SME). The SME should have a set of
best practices and candidate architectures that can be customized to fit the specific environment based on
the JAD session results and analysis.
A Software Development Methodology for EAI
Page 8
The Software Development Lifecycle Implications
The strengths of a modern SDLC comes into play during the analysis, design and construction phases
of an EAI effort. To consider the use of an SDLC for EAI, let us constrain the definition of EAI to include
building A2A and B2B integrations and not expand the scope to include Business Process Management
(BPM), portals, Business Activity Monitoring (BAM), and whatever else EAI vendors decide to throw into
their software stack.
The EAI effort then can be viewed as a subset of a larger effort to accomplish a business goal such as
building a composite application or a managing a business transaction using BPM with system integrations.
���������������
��������������������������
�����������������
���������������������������
����������������������� ������������
�������������������� ����������������
������������� ������������������
�������������������� ����������������������������
���������������
����������������������
�����
������� ������� �������
�����
������
�����
�������������
�������������������� ���������������������
����������������
�������� ������������������
����� ������
�����
�������
������� ����� �������
�����
����
A Software Development Methodology for EAI
Page 9
The larger effort may include user interfaces, portals, BAM, etc., but the EAI effort would be focused on
the system-to-system interfaces. The intent of defining EAI in this context is to focus on the process and
deliverables needed in this specialized area – purposely avoiding well understood topics such as user
interface design and business process modeling.
A2A integrations are a subset of the overall business process where the EAI architect analyzes the
interfaces and messages with fine granularly, but must understand the overall business process from the
perspective of transaction integrity, security and performance. This requires an understanding of the big
picture, but this can easily be accomplished by reviewing system level deliverables such as use cases and
activity diagrams.
By focusing on interfaces and messages EAI becomes analogous to OOA/OOD in that entire applications
are abstracted as objects. The goal is to create composite applications consisting of multiple applications
with messages flows between the applications interfaces (or adapters). The logical design objectives for
EAI consist of the following activities:
• Define the business domain to be integrated (e.g. new customer add)
• Define the integration points within the domain (e.g. a SAP adapter)
• Define the message flows and sequences between the integration points
• Define the message formats necessary for A2A interaction
• Identify the components that will be used for common services such as data mapping and error
handling, the message formats and component interactions
• Define the control logic necessary for message routing, transformations, etc.
• Define the test cases for the integration
The EAI design goal is to produce lightweight deliverables describing the interfaces, events and message
flows with an iterative methodology. UML proves to be well suited for EAI design artifacts since EAI treats
applications as objects and UML artifacts express object interactions. Consider the UML models and
deliverables in figure 3 in the context of the logical design deliverables for EAI as follows:
• Context diagram – defines the integration domain boundary
• Use cases – functional requirements
• Collaboration diagrams – component responsibilities and how they collaborate
• Sequence diagrams – message flows within the integration
• Activity diagrams – process flows through distributed EAI components
A Software Development Methodology for EAI
Page 10
The context diagram sets the logical boundary of the integration, essentially decomposing the goal of
EAI into iterative steps such as a business process, data element(s) or geography. The concept of an
integration domain is critical: Without decomposing the complex problem of EAI into manageable domains,
the problem becomes untenable. For example, consider the problem of defining the canonical form of a
customer in a global organization with hundreds of businesses and systems all over the world. One must
decompose this problem into domains with a domain integration strategy.
Within the context of the domain, use cases define functional requirements. Test cases are part of the
requirements that drive system design; users, analysts and developers must understand them. Test cases
are derived from use cases and event flows.
���������������
������������� ��������� ���������������������� ������������
�������� ���������������������
����� ��������� �����
�����������������
�����
����������������
���������������������
����������������
�����
�����
�����
����� �����
�����
������
�������� �����
����� ����������
A Software Development Methodology for EAI
Page 11
Collaboration diagrams are important to EAI to define the various components and their roles in the
integration – e.g. adapters, processes within brokers, mapping etc. Sequence diagrams further define
the event driven nature of EAI by depicting the message flows between EAI components in time-ordered
sequence (arguably the most valuable deliverable). Activity diagrams are also particularly useful for
showing the division of responsibility of an integration process across distributed systems in swim lanes.
The domain of the integration, as defined by the context diagram and use cases, establishes the domain
model for data integration. This is essentially the mapping of multiple data sources either point-to-point
between applications or to a canonical form. A rule of thumb is to use a canonical representation if five
or more systems will require the data. Canonical data formats then become part of the CIM. A simple
definition of the CIM (for brevity) is a common externally-managed repository of data semantics that
promotes reuse. The CIM is analogous to the vocabulary for future integrations and the UML interface
specifications (such as the sequence diagrams) are the way to communicate that vocabulary. The CIM and
interface specifications are critical for reuse and should be stored in an external repository – not buried in
broker process and mapping logic.
The notion of decomposing EAI to interface level granularity without introducing a higher level methodology
for a more holistic view of the end-goal of a totally integrated system seems counterintuitive. However,
while it is possible to understand EAI holistically at the interface level, it is difficult, if not impossible, to get
agreement on the desired holistic state of a totally integrated enterprise at the business processes level.
And, if you were able to capture all these business processes, it would simply be a snapshot in time since
business processes evolve quickly – EAI enables process agility. It is more affective to view EAI at the
application portfolio level with a strategic approach to improving high-value processes in priority sequence
– a role of the ICC. The methodology goal is to develop an adaptive EAI architecture using the following
techniques:
• Decomposition into integration domains with a domain integration strategy
• Encapsulate applications as objects with well defined interfaces to achieve low coupling
• Expand on the CIM with each integration domain, building the semantics of the enterprise iteratively
• Build the repository of interface definitions and message interactions for reuse with each integration
iteration
• Build the repository of design patterns (technical and business) for reuse
• Assemble composite applications based on application objects using orchestrated processes visually
modeled in the EAI tools
We have examined EAI Methodology in terms of a process and modeling language (UML). UML is the
graphical models that a methodology uses to convey system design. The methodology is a process that
defines the steps necessary to complete the analysis, design, construction etc.
A Software Development Methodology for EAI
Page 12
There is a continued debate on the merits of the RUP versus agile methods such as XP. While it is
certainly possible to run EAI projects in an XP fashion there is concern that the lack of rigor can lead to an
incomplete or inconsistent process around integration domains, the CIM and interface design specifications.
Since these deliverables become the basis for future integrations, process consistency is critical. We will
therefore look at a lightweight (i.e., consisting of few UML deliverables) instance of RUP as the candidate
EAI methodology – RUP’s iterative nature and focus on architecture early in the process are particularly
well suited to EAI.
The RUP architecture is defined in two dimensions as shown is figure 4. You progress through a project
timeline through iterations of the RUP defined phases. Within each phase the disciplines of RUP are
performed in varying degrees of effort as shown by the horizontal graphs. To consider the use of the RUP
for EAI we will map the RUP phases to EAI specific tasks and disciplines.
The RUP inception phase has the objectives of establishing the project vision, system functionality,
candidate solution, costs and schedule, and stakeholder final approval on the vision cost, etc. With respect
������
�����������
�����������������
������������
�����������������
������������������
��������������������������������������
�����������������������������
����������
���������
������� ������� ������� ��������
��������
��������
������
������
����������� ������������ ����������
A Software Development Methodology for EAI
Page 13
to EAI the RUP inception phase consists of the following activities:
• The ICC is engaged to prepare the EAI candidate solution
• A succinct project vision describes the problem, vision of the solution, stakeholders and high-level
use cases
• Non-functional requirements are assessed for architecture impact
• Artifacts are examined for reuse: CIM, interfaces specifications, components, etc.
• Candidate solution is proposed in terms of architecture, interfaces and EAI component roles
• The preliminary schedule and cost estimate is produced for approval
The RUP elaboration phase RUP has the objectives of creating detailed project requirements, the plans to
mitigate risks, the baseline architecture and refine project costs and schedule. A RUP elaboration phase for
EAI consists of the following activities:
• Create a risk list and action items to mitigate – e.g. technology pilot
• Complete UML deliverables for domain and interface design (figure 3)
• Complete domain specific message specifications reusing the CIM (figure 3)
• Detailed design of CIM, interfaces and configurations
• Define test strategy, plans and cases (figure 3)
• Create baseline architecture to meet non-functional requirements (figure 2)
• Refine project plans and costs
The RUP construction phase RUP has the objective of code construction while minimizing costs, creating
parallel tasks, and iteratively developing the complete solution. A RUP construction phase for EAI includes
the following activities:
• Create environments for test, QA and production
• Component builds to specifications and architecture
• Unit, integration and system test
• Stress and operational test – fault tolerance, capacity
• Create migration and build documentation
• Training for support and system administration
• Create monitoring and management tools
The RUP transition phase consists of user beta testing, user and support staff training, deployment and
user signoff. Also, the RUP documents lessons learned for future iterations. Specific to EAI, the RUP
transition phase consists of the following activities:
A Software Development Methodology for EAI
Page 14
A Software Development Methodology for EAI
Page 15
• Deployment and acceptance test
• Production turnover and user signoff
• Change management
• Environment management and operational support
• Update the CIM and interface repository
• Update design patterns and common components
The complexity of the RUP can be intimidating. However, one can construct a lightweight RUP instance for
EAI describing task flows, inputs, roles and deliverables in a single document – without buying the RUP
software. Also, the suggested UML deliverables can be produced with inexpensive modeling tools like Visio.
If the RUP is a corporate standard, configuring an EAI instance is straightforward.
Conclusion
The intent here is not to turn an EAI methodology into a study of academic purity or a religious crusade
(where zealots certainly abound) for an ideal approach, but to produce a practical and proven guide to
adapting EAI to existing practices. There is no perfect mapping of EAI to either the RUP or agile methods
like XP – especially in the early strategic stages suggested herein. Coming from the perspective of an
outside consultant, one must adapt to the environment to be successful with EAI. The approach outlined is
adaptable, lightweight, quick and, best of all, it simply works.
Takeaways
Business TechnicalA proven EAI methodology reduces risks and improves ROI
The Integration Competency Center should create a cross-functional team to provide the expertise, gover-nance and standards necessary for successful EAI.
The business case for EAI should explore opportunities for business process improvement and ways to generate revenue – not just the cost savings of improved integra-tion productivity.
The EAI architecture should be presented in multiple views – relevant to specific stakeholders.
An EAI methodology will create business process flex-ibility by allowing quick changes and the assembly of business processes from components.
The analysis and design deliverables for EAI should be lightweight using a visual modeling language such as UML to describe interfaces, events and messages.
The EAI methodology should be iterative with lightweight deliverables, but with enough rigor to develop and en-force enterprise standards and create an EAI repository for reusable artifacts.
Eric Roch is Vice President of Technology for ZettaWorks (a pure-play EAI consultancy), where he is responsible for
the development of EAI methodology and solutions. During his more than 20 years of experience in IT, he has worked
in roles such as executive-level management, technical architecture, and software development in leading technology
companies. Eric’s experience includes strategic planning and commercialization of methodologies and software, as well
as technical architecture for EAI and distributed applications.
e-Mail: [email protected]
About the Author
A Software Development Methodology for EAI
Page 16