MSIT: BizTalk 2013 R2 Migration StoryApplication migration from BizTalk Server 2010 to BizTalk Server 2013 R2

Arvind Chaudhary, Microsoft | Piyush Prasad, Microsoft12/1/2014

Copyright

This document is provided “as-is”. Information and views expressed in this document, including URL and other Internet Web site references, may change without notice. You bear the risk of using it.

Some examples depicted herein are provided for illustration only and are fictitious. No real association or connection is intended or should be inferred.

This document does not provide you with any legal rights to any intellectual property in any Microsoft product. You may copy and use this document for your internal, reference purposes.

© 2013 Microsoft. All rights reserved.

ContentsIntroduction.................................................................................................................................................3

Introduction to BizTalk Server 2013............................................................................................................3

New Features of BizTalk Server 2013..........................................................................................................3

Microsoft IT (MSIT) BizTalk Server 2013 Migration Story............................................................................4

Problem Statement.................................................................................................................................4

Microsoft IT Reasons for Migrating to BizTalk Server 2013.....................................................................4

Migration Process....................................................................................................................................5

Development Strategy.........................................................................................................................5

Step 1: Identify Message Stream and dependencies...............................................................................5

Step 2: Visual Studio Project Migration...................................................................................................6

Step 3: TPM (Trading Partner Management) Migration..........................................................................9

Step 4: BizTalk Application Binding Updates.........................................................................................10

Testing Approach...................................................................................................................................11

Challenges faced in Migration...............................................................................................................11

Best Practices........................................................................................................................................12

Appendix...................................................................................................................................................13

Acknowledgements...............................................................................................................................14

IntroductionThis white paper describes the following:

Motivations for moving to BizTalk Server 2013 R2. The process that Microsoft IT went through to migrate critical integrations from BizTalk Server

2009/2010 to BizTalk Server 2013 R2.

Introduction to BizTalk Server 2013To run any business in today’s world, there are multiple Enterprise Business processes that must be supported within an organization. These processes are typically dependent on multiple systems working together to achieve certain goals. Demands for integration have grown and include:

Faster plug-n-play architecture Be ready and capable to support hybrid solutions with cloud applications

BizTalk Server 2013 R2 provides the right set of features to enable enterprise mobility. Also, migrating/upgrading to BizTalk Server 2013 R2 is easier.

New Features of BizTalk Server 2013 R2BizTalk Server 2013 R2 introduces enhancements and new features in the following areas:

For details on new features, see:http://msdn.microsoft.com/library/jj248703(v=bts.80).aspx

Microsoft IT (MSIT) BizTalk Server 2013 Migration StoryProblem Statement

With demand on Microsoft Business to be agile and scalable, Microsoft IT decided to move all its operations and services to the cloud. The goal of the integration team is to enable this transition seamlessly with zero business impact.

With mainstream support for BizTalk Server 2009/OS/SQL ending soon, this migration needs to be done to stay current.

Microsoft IT Reasons for Migrating to BizTalk Server 2013 R2 Virtualization – Companies are focused on cost reduction. Companies are moving from data

centers (and all the maintenance associated with data centers) to efficient and cost-savings methods provided by cloud offerings. Virtualization is the key factor to moving to a cloud-based infrastructure.

Hybrid Solutions using Azure Service Bus/MABS – With mobility, more and more services are available on the cloud. Hybrid solutions that include cloud and on-premises resources are key to businesses. High scalability, availability, and throughput make the cloud the way to go.

Enabling Service Bus Features – On-demand integration is now available out of the box with support of ESB Toolkit features.

Simplified manageability – BizTalk Server 2013 R2 offers a simplified administration console making enterprise messaging easier to manage and support.

Stay Current – Being able to use newer communication protocols makes it easy to stay current and take advantage of new software features and technologies. Mainstream Support for Windows Server 2008 R2 and SQL Server 2008 R2 are also key. To keep active with support for mission-critical IT assets, it is essential to upgrade to the latest versions.

Migration ProcessMSIT Development Strategy

Step 1: Identify Message Stream and dependenciesIt is essential that you create a repository of information on all the applications you plan to migrate. MSIT started the process by estimating the number of BizTalk applications and integration flows in each application. Care was taken to understand the application traffic and revisited architecture. Between BizTalk Server 2009 and BizTalk Server 2010/2013 R2, there are new features (like out of box ESB Toolkit) added; which can potentially optimize performance if you are using custom components in place of the Toolkit. Also, MSIT optimized applications by creating a pure messaging solution wherever orchestration was not required. Before you migrate, MSIT recommends doing the same.

After a well-executed performance cycle, it was decided to host new BizTalk Server 2013 instances on virtual machines. The following topology diagram describes one such instance:

Identify Message

Stream and dependencies

Visual Studio Project

Migration

Trading Partner

Migration

BizTalk Application

Binding update

One of the important steps is to identify the criticality of the application. Start with applications of low complexity and low criticality; later move to medium to high complexity applications. Complexity can be defined on the following parameters:

a. Application Architectureb. Business Impactc. Message Traffic

The project approach is based on Scrum Agile Methodology . As previously mentioned, applications are divided into flows. During Sprint planning, we started pulling these stories from product backlog based on the team capacity. Stories were split based on these flows.

Step 2: Visual Studio Project MigrationOnce flows have been identified for the migration sprint, the BizTalk solutions are rebuilt using Visual Studio 2013.

Address the warnings and comments given by Visual Studio

Immediately, you see the following warning:

By default, the one way migration changes your .NET version to 4.5. If there are any dependent components that do not work on this .NET Framework version, take the necessary steps to change the component to a relevant version.

The following image is a sample migration report that is generated. Note the warning:

We made a few more changes on the BizTalk solution while migrating:

a. BizTalk solutions that contained legacy adapters, like SAP and SQL, are replaced with WCF adapters and their bindings in BizTalk Server 2013 R2.

b. Due to the growing number of custom BizTalk pipelines, all the custom pipelines are replaced with pipelines using base components built with an internal framework called Pipeline Consolidation Framework (PCF). The following image is snapshot of PCF XML pipeline configuration:

The framework offers flexibility for pipeline configuration, minimizes the number of BizTalk pipelines, centralizes the storage in the .NET global assembly cache (GAC), and makes the pipeline component reusable with more shared functionality. Refer to this blog for more information on PCF (http://blogs.msdn.com/b/srikiran/archive/2012/12/22/pipeline-consolidation-framework-pcf-biztalk-2010.aspx).

Changes to the existing solutions are done when configuring adapters, pipelines, and party settings. In other artifacts, like maps, schemas and orchestrations, no changes are required. The final solutions are built and tested in the required environments.

Step 3: TPM (Trading Partner Management) Migration This step applies if you are doing business-to-business (B2B) integration and using BizTalk parties. We maintain all EDI X12/EDIFACT protocol versions using BizTalk’s out of the box functionality to create Party and Agreements. There a large number of existing trading parties in BizTalk Server 2010 applications. The challenge is to import the party configuration without missing any key data in the new BizTalk Server 2013 R2 environment.

The BizTalk Server 2013 R2 TPM database introduces new fields for X12 agreements; which block the direct import of party data. If you export party bindings from BizTalk Server 2010 and then import them into BizTalk Server 2013 R2, the following error can occur:

This error is specific to X12; EDIFACT party import works as-is. Since the number of trading parties and agreements are large, the team focused on finding the gaps between BizTalk Server 2010 and BizTalk Server 2013 R2 TPM details.

Comparing the exported party data from both versions of BizTalk (BizTalk 2010 and BizTalk 2013 R2), the following fields are added in X12 Agreements in BizTalk Server 2013 R2:

FunctionalAckVersion BatchImplementationAck GeneratePatAK901 GroupHeaderVersion ImplementationAckVersion NeedImplementationAck TrimLeadingAndTrailingSpacesAndZeroes

All tables starting with <tpm.X12> are analyzed to locate the extra fields added in the new TPM. The following tables changed in BizTalk Server 2013 R2:

1) Tpm.X12ProtocolSettingsa. FunctionalAckVersion ([nvarchar](5) NULL)b. NeedImplementationAck ([bit] NOT NULL) c. ImplementationAckVersion ([nvarchar](5) NULL)d. BatchImplementationAck ([bit] NOT NULL)e. GeneratePATAK901 ([bit] NOT NULL)f. TrimLeadingAndTrailingSpacesAndZeroes ([bit] NOT NULL)

2) tpm.X12ValidationOverridesa. TrimLeadingAndTrailingSpacesAndZeroes ([bit] NOT NULL)

To ensure smooth and error-free migration of TPM from BizTalk Server 2010, the Party Migration Tool was created (available here). Use this tool if you see this error while importing X12 Agreements.

To use the tool, export the party binding from BizTalk Server 2010 and use this binding as input. The migration tool:

Updates the relevant data in the X12 agreements. Generates another party binding that can be imported into BizTalk Server 2013 R2. Adds default values to the extra fields previously listed.

Step 4: BizTalk Application Binding Updates The final step is to update the BizTalk application bindings with the new adapters/pipeline changes, as applicable. MSIT made the following changes while migrating:

1. WCF adapters are used wherever applicable.2. All pipelines components are replaced with the PCF Framework.3. The custom adapter for the cloud is replaced with the SB-Messaging adapter included with

BizTalk Server 2013 R2.4. Custom behavior to support REST is replaced with the REST adapters included with BizTalk

Server 2013 R2.5. Any gateway is replaced with Azure Service Bus utilizing the SB-Messaging Adapter included

with BizTalk 2013 R2.

Testing ApproachTesting is done in multiple stages. As most of the testing aspects (like Build Verification Testing, Environment Testing, and Functional) are repeated across environments, automation frameworks are used along with continuous integration builds [Build Deploy Test] for Cycle Time reduction. We recommend using a process similar to the following:

1. Unit Testing – BizUnit or Map Test Framework (MTF) is used to verify the new map logic.2. Developer Integration Testing – Provides limited comparison of output between the two

environments (old and new).3. Build Verification Testing – The build is verified using the binding; which includes ports, parties,

agreements, configuration files, assemblies, versioning, and deployment scripts. As the testing is reused across environments, an automation framework is used to validate the configurations based on XPaths.

4. Environment Verification Testing – Checks the dependent component installation and topology.5. Functional Testing – Business flow testing, adapters, pipeline configuration, tracking, monitoring

and alerting are verified.6. Performance Testing – Based on performance test requirements, the old translator is

benchmarked. Tools like LoadGen, Performance Monitor, and Performance Analysis of Logs (PAL) are used to simulate scenarios, capture counters, and analyze the results.

7. User Acceptance Testing (UAT) – Testing and validation by end users (external partners and internal business teams) on a pre-production environment.

8. After and Before (AB) Testing – Bulk Data Comparison testing of at least one month of data output between the two environments (old and new). This ensures better coverage of the execution paths. In case of low frequency transactions, the archival is done for an extended period to get more test data.

9. Rollback Testing – For a seamless roll back to the 'Last Known Good Migration' back to the old system, rollback testing is done in SIT (System Integration Testing) and UAT phases connecting to downstream and upstream systems.

10. Security Testing – For sensitive data transactions, the appropriate security tests are executed at various stages of transmission. Data sanitization measures are taken to scrub and mask the production data before being used in test environments.

Challenges faced in Migration There are some challenges faced during the migration of applications to BizTalk Server 2010. The most important include:

1. To ensure the seamless migration, the team recommends AB testing; which does comparison testing between new and old output.

2. As an integration team, ensure upstream and downstream systems are involved in integration testing on the new environment.

3. Any architecture changes need to be thought through to ensure the process is not breaking.

4. Documentation might not be available, in which case you need to reverse engineer flow and application information using Binding.

5. Business documentation should be updated post-migration.

Best Practices To help the migration process go smoothly, use the following best practices:

Ensure consistent monitoring and tracking of messages is present across all applications. Avoid overloading individual BizTalk Server virtual machines. Instead, use the appropriate

number of BizTalk Server 2013 R2 virtual machines and decide how many applications to deploy on each server in the design phase.

Validate your message volume per host and modify the default host settings as appropriate. Ensure that the BizTalk SQL Server Agent jobs run successfully. Ensure you have a backup and replication strategy in place. When possible, avoid large schemas with mostly optional fields. Modify default retries and timeouts as appropriate.

AcknowledgementsThank you to the reviewers and contributors, including:

Saurabh Deo, Microsoft Nikhil Tayal, Microsoft Anil Chandra Lingam, Microsoft Rajesh Ramamirtham, Microsoft Soumitra Kumar Dhar, Microsoft Mandi Ohlinger, Microsoft Atin Malik, Microsoft Banupriya, Microsoft Sumant Kumar Pandey, Microsoft

Did this paper help you? Please give us your feedback. Tell us on a scale of 1 (poor) to 5 (excellent), how would you rate this paper and why have you given it this rating? For example:

Are you rating it high due to having good examples, excellent screen shots, clear writing, or another reason?

Are you rating it low due to poor examples, fuzzy screen shots, or unclear writing?

This feedback will help us improve the quality of white papers we release.