BizTalk Value

August 2006

Alan Smith

About Alan Smith

BizTalk Consultant• Long history of Microsoft development• Working for KnowIT Consulting in Stockholm• Architect, developer, and mentor roles

BizTalk Trainer• Delivering BizTalk Server courses at Informator• Delivering advanced “BizTalk Deep Dive” courses for QuickLearn

BizTalk MVP• Active in BizTalk Server developer community• Publish “The Bloggers Guide to BizTalk”• Run BizTalk User Group Sweden (BUGS)

BizTalk Server Value

• BizTalk server overview• BizTalk in the Microsoft product line

BizTalk Server Introduction

• Publish subscribe message hub• Integration server• Business process workflow engine• Business intelligence tools

BizTalk Feature Set

• The importance of integration• Problems implementing integration

System Integration Overview

• Batch oriented• Message oriented• Service oriented

Integration Strategies

• FRAS Integration hub

Case Study

BizTalk Server Timeline•Artifacts developed using

numerous toolsBizTalk Server 2000

•Improvements made to several features

BizTalk Server 2002

•Complete rewrite of product•Built on .net 1.1•Visual Studio Development Tools•Massive improvements in all areas

BizTalk Server 2004

•Built on .net 2.0Big improvements in massaging engine

•New Administration console•BAM Improvements

BizTalk Server 2006

•Integration with WCF and WF•Integrated support for RFID devices•Support for EDI documents and AS2 transactions•Integrated support for office 2007 and SharePoint

BizTalk Server 2006 R2

•Includes “BizTalk Solution Designer?•Includes new BizTalk mapper

interface?BizTalk Server

V Next

BizTalk Server 2006 Licensing Options

Enterprise Edition

• $29,999 (210,000 SEK) per processor

• Complete EAI, B2B, and Business Process Management functionality

• Includes all current and new application and technology adapters

• Unlimited scale-out, scale-up and clustering

• Includes Host Integration Server Enterprise Edition

Standard Edition

• $8,499 (60,000 SEK) per processor

• Complete EAI, B2B, and Business Process Management functionality

• Includes all current and new application and technology adapters

• Limited to two CPUs on a single server

• Limited to five internal applications

• Includes Host Integration Server Enterprise Edition

Developer Edition

• $499 (3,500 SEK) per user

• Complete EAI, B2B, and Business Process Management functionality

• Includes all current and new application and technology adapters

• Included in MSDN Universal Subscription

• Limited to designing, development and testing solutions

BizTalk Server in the Microsoft Product Line

BizTalk Server Development

VisualStudio.net 2005 is used as the BizTalk development environment

BizTalk artifact development is typically performed using graphical interfaces

As BizTalk Server is built on .net, the development process shares many similarities with a .net project

BizTalk Server functionality can easily be extended using C# or VB.net

BizTalk Sever Administration Tools

BizTalk Server 2006 provides greatly improved server administration

The BizTalk Server Administration console can be used to administer a multi server BizTalk deployment

Powerful command line tools are available for automated script driven deployment

BizTalk Server Feature Set

• Business Activity Monitoring (BAM) provides real time and aggregated views of business data

• Business Rules Engine allows complex business decisions to be abstracted form the business process

Business Intelligence Tools

• BizTalk Orchestration Engine supports complex business processes• Persistence of orchestration state enables long running transactions and robust

recovery from system failures

Business Process Workflow Engine

• Adapters provide communication with other systems• Protocols (FTP, SMTP, POP3), databases (SQL Server, Oracle, DB2), applications

(SAP, Tibco, SharePoint, Siebel)

Integration Server

• Message box database implemented on SQL Server• Strong support for guaranteed delivery• Configurable message tracking

Publish Subscribe Messaging Hub

Publish Subscribe Messaging Hub

Publish Subscribe Messaging HubPublish Subscribe Architecture• All messages that are received are published into the message box database• Each message can have zero, one, or more subscribers• Subscriptions can be based on message context (e.g. where the message was received

from) or message content (e.g. the name of the company supplied in a message body field)

“Guarenteed Delivery”• All messages are persisted in MessageBox database, ensuring recovery in the event of

severe failures• Retry logic and backup transports are provided out of the box when sending messages

Highly Scalable• A BizTalk group can easily be scaled to multiple servers• Server group can be tuned for high throughput, or low latency

Highly Available• Multi-server environments can feature redundant components to provide high availability

Error Handling• Handling of errors in message processing is configurable, and supplied out of the box

Integration Server

Adapters• Adapters provide communication between BizTalk Server and external

systems• Using adapters abstracts the complexity of communicating with a system

from the developer• BizTalk Server ships with a wide range of basic adapters out of the box• A number of enterprise application adapters are included within the

BizTalk Server license• Third party companies deliver a wide range of adapters for connecting to

line of business (LOB) applications• BizTalk Adapter Framework allows for the creation of custom adapters

BizTalk Server 2006 R2• Adapter framework to be based on Windows Communication Framework

(WCF), formally known as Indigo

BizTalk Adapters

Out of the Box

• File• FTP• HTTP• SOAP• POP3• SMTP• Windows

SharePoint Services• SQL Server• Base EDI• MSMQ• Websphere MQ

Included in License

• SAP• PeopleSoft

Enterprise• JD Edwards One

World XE• ODBC Adapter for

Oracle Database• Siebel eBusiness

Applications• TIBCO Rendezvous• TIBCO Enterprise

Message Service

Included in License, Available Soon

• Host Applications• IBM DB2• Host Files

Business Process Workflow Engine

Long running business processes can be implemented using Orchestrations

Graphical design environment provides for rapid development

Visual representation of process aids in communication

Orchestrations are persisted to SQL Server database to provide high availability

Orchestration engine provides a sophisticated transaction ad error handling model

Seamless integration to .net code

Business Intelligence Tools

BAM

• Business Activity Monitoring

• Provides powerful views of information processed by BizTalk

• Information in intercepted, business processes are not affected

• Information can be viewed in real time, or aggregated using SQL Server Analysis Services

BAS

• Business Activity Services

• Allows management of configurable trading partners

• Partners can be added, and modified using a SharePoint interface

• Custom agreements between partners can be created

Business Rules

• Provides a forward chaining inference engine

• Business decisions can be created and modified whilst system is running in production

• Delivers high performance, and allows generation of complex rule sets

Business Activity Monitoring

Business Analyst

Review business process and

message contents

Decide which information is

relevant to business process

Decide how that information should

be viewed

Developer Develop BAM artifacts

IT Professional

Deploy and configure BAM

artifacts

Business Analyst

View data provided by BAM

Configure and receive alerts

What is Integration?

”A set of technologies that enable organizations to control the quality of information by ensuring that it remains complete, consistent, and current across the entire portfolio of applications that define the operations of the business” - IDC

Integration Adds Value

Integration might provide substantial added values …• Support for processes vs. /just/ functional

support• Previous investments are secured• Increased revenue • Lead time & cost reductions• Better focus on the core business• New business models

The Importance of System Integration

Cost Savings• Reuse of existing architecture and systems• Reduced IT, personnel, and training costs

Information Flow• Transfer of information between systems• Maintaining up to date information across diverse applications in

an enterprise• Automate manual processes

Acquisitions and Mergers• When companies merge or acquire other companies, information

must be shared• Integration provides an unobtrusive method of achieving this

Problems with System Integration

Cost• Integration systems are expensive to purchase• Custom written solutions can be expensive to develop

Complexity• Integration is, by nature, a complex discipline• Knowledge of multiple platforms, applications, and protocols

is required• Often consultants from multiple companies will be required

Project Overruns• Complex development can often lead to overruns• This affects the companies budget, and business operations

How BizTalk Server Addresses These Issues

Cost• BizTalk Server compares very favorably with other integration systems• BizTalk Server Standard Edition is just $8499 (60,000 SEK)

Complexity• BizTalk Server brings integration development within the reach of .net

developer• The use of adapters abstracts much of the communication

implementation from the developer• Windows Server and SQL Server admin and maintenance skills can be

leveraged to maintain a BizTalk deployment

Project Overruns• With adequate knowledge, BizTalk Server provides a highly productive

development environment• Powerful graphical tools can lead to rapid application development

Integration Strategies

Batch Oriented

• Systems can output and input batches of information

• Batches are usually text based files

• Batches are transferred according to a fixed time interval (daily, weekly etc.)

Message Oriented

• Systems exchange messages with each other

• Messages contain individual objects, e.g. one order, one stock quote

• Messages are often transferred using asynchronous message queues

Service Oriented

• Web services are exposed by systems

• Other systems call these web services

• Standard protocols are used

Batch Oriented Integration

The source system will output a batch of information to a file

The batch exports are performed according to a specific time schedule, typically daily, weekly etc.

The batch files are typically text based flat files

The target system will import and process these files to update its information

Batch Oriented IntegrationInformation is present in System A

Batch Oriented IntegrationA batch file is exported from System A

Batch Oriented Integration

Batch Oriented IntegrationThe batch file is imported into System B

Batch Oriented IntegrationInformation is present in System B

Disadvantages with Batch Oriented Integration

Information is only updated during each batch, so there are often times when it is out of date

Batch files can become very large, and take time and resources to process

The flat file format of batches can be complex when a hierarchical structure is required

Robust error handling is required to ensure information consistency

Day 1

Information present in System A

Day 1

Information present in System A

Day 1

Information present in System A

Day 1

Information transferred to first intermediary system

Day 2

Information transferred to first intermediary system

Day 2

Information transferred to first intermediary system

Day 2

Information transferred to second intermediary system

Day 3

Information transferred to second intermediary system

Day 3

Information transferred to second intermediary system

Day 3

Information present in System B

BizTalk Server Batch Support

BizTalk Server converts text based files to and from XML

Batch files can be split into individual messages as they enter BizTalk Server

Individual XML messages can be aggregated to produce a batch file

BizTalk provides powerful error handling and robust recoverable interchange processing to handle errors in a message batch

Much of this functionality can be configured at run time

The Flat File Schema Wizard provides a powerful graphical tool to convert complex flat file structures to and from XML

Splitting a Batch File into XML Messages

Aggregating XML Messages to Send as a Batch File

Demo: Batch Processing in BizTalk Server

In this demo you will see:•Converting a text batch to XML messages•Handling errors that occur in a batch

Message Oriented Architecture

Systems communicate in real time by sending messages asynchronously

M essages contain “Byte Sized” pieces of inform ation, e.g. an order, an invoice payment

Messages are often transferred using asynchronous queuing applications (MSMQ, WebSphere MQ, BizTalk Messaging)

Messages are often sent as XML, and based upon industry standards

Advantages of Message Oriented Integration

Speed of Data Replication• Messages are typically delivered across multiple application

boundaries in seconds, rather than days• Errors can be detected and fixed more quickly

Ease of Implementation• Byte sized pieces of information are easier to process than batches

Use of Standards• XML standards are clearly defined• XSD schemas can be used as message definitions for multiple

systems

Enterprise Integration Patterns

Industry standard pattern language for message based systems

Provides detailed descriptions and implementation guidelines

Downloadable appendix describes BizTalk Server 2004 implementations

Enterprise Pattern Implementation in BizTalk Server

Out of the Box

• Message Endpoint• Publish-Subscribe

Channel• Guaranteed Delivery• Process Manager• Polling Consumer• Event-Driven Consumer

Easy to Implement

• Message Translator• Message Router• Point-to-Point Channel• Invalid Message Channel• Dead Letter Channel• Channel Adapter• Correlation Identifier• Content-Based Router• Message Filter• Dynamic Router• Recipient List• Splitter• Envelope Wrapper• Content Enricher• Content Filter• Normalizer• Wire Tap

Harder to Implement

• Aggregator• Resequencer• Scatter-Gather• Test Message

Service Oriented Architecture (SOA)

Platform independent

Based on open industry standards

Massive vender support

Message based communication

Brings reusability to systems and applications

Flexibility for designing system integrations

Problems with SOA

Legacy systems may not provide SOA support

Implementing SOA for these systems may be difficult and expensive

Lack of real SOA architecture and development knowledge

Adding SOA to Legacy Systems

• Interfacing with a legacy system can be complex and require special knowledge

• Management of security of authentication can be complex

• Long running maintenance can be required

Build a service layer onto a legacy system

• BizTalk Server exposes a SOA interface to clients• Adapters can often provide interface to a legacy system• Security can be managed using Enterprise Single Sign On

Use BizTalk Server to add a SOA layer

BizTalk and SOA

BizTalk Server easily integrates with SOA

Web services can be consumed or exposed by BizTalk Server

BizTalk Server provides a rich range of adapters, making it an ideal bridge between SOA and legacy systems

BizTalk Server provides highly reliable messaging and message tracking out of the box

BizTalk and SOA

BizTalk Server can act as an Integration Enabler exposing the functionality of legacy systems as a web service

Communication between BizTalk and legacy systems is implemented using BizTalk adapters

Integration Enabler pattern taken from “The Arch Hackers BizTalk Blog”

Business Process Engine

Often, communication between systems is more complex than sending a message

Business processes can be required involving complex rules, and long running transactions

The BizTalk Server Orchestration Engine provides a platform for implementing complex workflows

The Business Rules Engine allows configuration, management, and deployment of complex business rules in real time

Business Activity Monitoring (BAM) can provide real time feedback of business process execution

BPM + Integration

Message Oriented Integration• “BizTalk Server is passive w ith respect to the business processes,

and acts as a service to route messages between systems. Messages may be tracked, but no business relevant processes are im plem ented in the m essage engine.”

Business Process Management• “BizTalk Server is active w ith respect to the business process.

Business processes are implemented within BizTalk Server, and consume the services provided by other systems. Business process decisions are implemented using the Business Rules engine, and business activity monitoring is used store business relevant information and provide users with a view of the business processes.”

Demo: Business Process and Workflow

In this demo you will see:• Implementation of human based workflow in BizTalk Server• Integration with Microsoft Office InfoPath• Integration with Windows SharePoint Services

Human Workflow in BizTalk Server

The combination of using BizTalk Server,

InfoPath and SharePoint allows

basic human workflow to be

implemented in the business process

BizTalk Server

SharePointInfoPath

“InfoPath is to BizTalk what Outlook is to Exchange”

Case Study - FRAS

Creating new opportunities in the Dairy business

FRAS – Business Objectives

•Using a common transport capacity (logistics)•Selling dairy products nation wide•Each company assuming at least two different roles•Connecting each dairy’s ERP system in a reliable and secure fashion•Exchanging business information with a maintained customer integrity

The Swedish dairies launched a new business model in 2005 that needed to support:

•An integration solution •Support for various message formats•Support for various business messages•Support for future changes without a complete rewrite

This called for:

•Relies on Microsoft BizTalk Server 2004 as the integration mechanism •Provides a communications infrastructure that allows for future growth•Provides web-access to operational status for critical information (stock,

invoice, order etc.)

The solution:

Processes and operations

Distribution – Supply Chain

Brand owner Carrier Customer

StoreDistribution Warehouse

DairyWarehouse

FRAS – Process overview

Support processes

Planning

OrderDelivery

Refill

Planning

3

VM-ägarrollen Transportörsrollen

Fakturering på vanligt sätt

Uppdatering av artikelregister med VM-ägarens artiklar.

Uppdatering av kundregister med Transportörskunder

Följesedelinfo VM-artiklar

Hubb

Överföring av förhandsorder från säljväskor

Påfyllnad av lager

Uttag från lager

Inventering

Reklamation

Hanteringssvinn

Godsmottagning

Kunden

Orderinfo VM-artiklar

Uppdatering av VM-ägarens artikelsortiment per kund.

”H u b b en ” m öjlig gö r tran spo rtsam arb etet m ellan m ejerifö retag en g en om överföring av information

och sammanknytning av ERP-systemen:

Från VFrån T

4

1

2

3

5

6

7

8

9

10

11

12

FRAS generellt

Fakturor/Samlings-fakturor(Central-

fakturering)/Gemensam

fakturaOrder13

1415

Fsg-stat

Fsg-stat

ERP ERP

FRAS – Technical profile • Hub – BizTalk Server 2004– Development tools BTS / VS

C#– two physical servers for

BizTalk, two servers for SQL Server 2000

• VPN tunnels• Exchange

– IBM/ MQ Series – FTP

• Asynchronous connections with a high overall throughput / performance

• Functionality provided:– Logging– Transformation– Synchronization– Routing– Persistence (customer & line

items)

Sveriges Mejerier

• A new business model• An efficient re-use of existing logistics- and IT-solutions• An option to scale out, and up, without having to refrain to a com plete “rew rite” – a flexible and dynamic solution

• A comparatively low cost for the IT part of the solution (approximately 13 MSEK including maintenance until 2007)

The Net Result?

Lessons learned- Experiences

- Pitfalls- Recommendations

• Integration requires many ”disciplines”…– Project manager

– Test experts

– O perations’ experts

– IT Infrastructure

– Developers

– Integration platform experts

– …

• Relying on an integration platform is almost always a profitable investm ent…– But there are alternatives:

• Data exchange (via DTS, SSIS or equivalent)

• Custom developed integration

The road to successful integration?

We recommend – using an EAI-tool for

• Process support and development

• Business activity monitoring (BAM)

• Application integration

– relying on

• O pen standards (if possible… )

• An Integration Factory™

• Proof-of-concept in the early phases

• Best practices

• Composite & Enterprise Integration Patterns

In the Near Future…

BizTalk Server 2006 R2• Integration with .net 3.0 technologies (Windows Workflow

Foundation, Windows Communication Foundation)• Integrated support for RFID devices• Support for EDI documents and AS2 transactions

RFID Services Platform• "We are committed to this space because it's an important

technology for our customers and our partners, We're committed to providing the technology that our customers want and that our partners need to RFID-enable their solutions. You'll see us bringing innovation to this space." - Javed Sikander - Microsoft program manager for industry solutions

Going Further…

• www.microsoft.com/biztalk - BizTalk homepage• msdn.microsoft.com/biztalk – BizTalk developer center• “Understanding BizTalk Server 2006” – Dave Chappell - introductory white

paper on BizTalk Server 2006 available on msdn.microsoft.com/biztalk• www.biztalkgurus.com – BizTalk community site• BizTalk Server 2006 Virtual Labs on MSDN• “The Bloggers Guide to BizTalk” – Free e-book containing over 350 articles

On the web

• 5 day developer course (MOC)• 2 day administration course (MOC)• 5 day advanced “BizTalk Deep Dive” (Q uickLearn)• 5 day BizTalk & SOA course (QuickLearn)

Courses

• SOA & Business Process Conference - October 3rd to 6th - Redmond

Conferances

Contact Information

• Blog: geekswithblogs.net/asmith