Exploring Data Integration PatternsLearn about the four key data integration patterns and the methods for putting them into action. With the Dell Boomi Platform, you can quickly build and easily manage all your integrations.

2 eBook | Integration Patterns

3 Introduction

4 Integration Patterns

8 Using Patterns to Solve Integration Challenges

13 Accelerate Your Business With a Unified Integration Platform

Table of Contents

3 eBook | Integration Patterns

There’s no question that in today’s business world, you need a solid technology foundation to succeed. New SaaS applications hit the market every day. And established enterprises are migrating to a hybrid IT environment to leverage these best-of-breed cloud applications.

Even for businesses taking a cloud-first approach, IT complexity can be baffling. And almost inevitably, this results in data silos that block customers, employees and partners from getting the data they need when they need it. To make information, interactions and innovation flow faster inside and outside your organization, you need integration.

But integration is full of challenges. For anyone in technology, the phrase “disparate systems” will ring a few bells. Different data formats. Different languages. Multiple operating platforms. Decentralized systems — or centralized systems that aren’t flexible enough to support a decentralized working model. To this, add an ever-increasing volume of data and rapid technology innovations.

When dealing with these challenges, enterprise architects frequently come across the same problems, which fall into a handful of common integration patterns. In almost any integration project, one or more of these patterns will be the way you want to build your application and data integrations.

In this ebook “Exploring Data Integration Patterns,” we’ll cover the top four integration patterns and the methods for putting them into action. And we’ll explain how you can quickly build and easily manage any of those integrations with the Dell Boomi Platform.

Introduction

4 eBook | Integration Patterns

Section 1: Integration Patterns

5 eBook | Integration Patterns

There are four common data management techniques that enterprise architects use to address integration challenges. Each has advantages and drawbacks. It’s important to understand all of them to know the best one — or combination — that will provide the most effective support for a given data integration project.

1. Transformation2. Aggregation3. Routing/Mediation4. Orchestration

A handy mnemonic for this is “taro,” like the starchy root vegetable.

1. Transformation

Different systems and applications have different data types and different data formats. For one system to accept data from another, changes need to be made.Data transformation is the process of converting data or information from one format to another. For example, if you want to run analysis of structured and unstructured data, you need to join these as a unified dataset. Another kind of transformation might be to convert data from a CSV file to XML, so it can be opened with the right tools.But data transformation goes beyond simple compatibility conversions. The transformation process can also include data cleansing, such as deduplication of records, and data enrichment, like appending data through aggregation.

Data Integration Processes

XML

Source/Target Source/TargetTranslator

JSON

6 eBook | Integration Patterns

2. Aggregation

Aggregation is important for eliminating data and business silos. It brings together the data you need to create a comprehensive view.Say you have five, six, ten or more applications that need to talk to one another. With aggregation, you can extract and process data from multiple systems, such as compiling customer data from Salesforce, Workday and SAP, and presenting a single view to your customer service reps. Then you can deposit it in a single data repository like a data warehouse or data lake.But since each aggregation is specific to each request, additional lines of code are required. Building the back-end logic for each new request quickly becomes unwieldy.

3. Routing/Mediation

Data routing is a critical part of building an integration process. When a request arrives, you need to understand where it should go — where to route it. You also need to know what data is needed.For example, you may have a support ticketing system where you want to trigger certain behaviors for certain customers. If a customer has paid extra for platinum-level support, you want to ensure its service calls are given priority and routed to senior support staff.To ensure the priority customer’s support ticket gets to the right people in a timely manner, you need rules or logic (mediation) that use metadata to appropriately route that request.

DATA INTEGRATION PROCESSES

Source

Source

TargetProcessingLogic Source Processing

Logic

Target

Target

Target

7 eBook | Integration Patterns

4. OrchestrationBuilding a complex response composed of data from multiple sources and systems requires data orchestration. Data orchestration facilitates and tracks data flows and data consumption from disparate sources, within a strong data governance framework. This goes beyond simple aggregation. It involves creating an automated process or workflow to fulfill a specific requirement. Data orchestration works a bit like a conductor but for automating and streamlining processes involving multiple systems. Let’s take invoice creation as an example.When a sales order comes through, there are a few processes, all of which probably happen within separate systems:

• The invoice needs to be generated.• The details of the order need to be filled in.• Warehouse systems need to be updated.• The invoice needs to be sent to the

purchaser.And later, the order needs to be fulfilled, payment applied to the invoice, a receipt sent, etc. Orchestration makes sure all these processes happen in the right order, at the right time.

DATA INTEGRATION PROCESSES

TargetTarget

Notification

Source

Target

ProcessingLogic

8 eBook | Integration Patterns

Section 2: Using Patterns to Solve Integration Challenges

9 eBook | Integration Patterns

It’s a given that you need to integrate your data. But you still have choices about how to make that integration happen. You must decide on your approach to data mapping. Next, you need to aggregate the data. And then transform it.With a clear understanding of the four types of integration patterns, let’s look at the best ways to put them to use.

1. Point-to-Point

The most common integration process is point-to-point — simply connecting two applications as quickly as possible.Let’s say you need to send an update from Salesforce to NetSuite. With a point-to-point connection, you take the data from one place, in this case Salesforce Sales Cloud, and put it into another — a NetSuite ERP. It’s that straightforward. You can use point-to-point connections for either real-time or batch integration, and they can be bidirectional. These integrations provide a quick

and easy way to replace manual data entry processes and speed workflows. But it’s a tactical, rather than a strategic, approach. Very few (if any) companies have only two applications to connect. With the volume and complexity of data, even small businesses have to be good at managing their data and creating unified views of their business.Because point-to-point builds each connection separately, it’s not scalable. When you have more than two endpoints, point-to-point fails.If you try to connect multiple point-to-point integrations, you wind up with “spaghetti code.” Every time something changes in one application, you have to adjust the code in every integration – individually. This complex, tangled pile of code is difficult and time-consuming to maintain. And it slows down your business.

Data Integration Patterns in Action

Target SourceProcessingLogic

Translator

10 eBook | Integration Patterns

2. Publish/Subscribe

Publish/subscribe messaging, commonly referred to as “pub/sub,” is perhaps the most well-known integration model.It uses asynchronous service-to-service communication, where senders publish messages categorized by specific topics. Data is stored in a queue, and each receiving application consumes that data at its own pace. Queuing mechanisms can help govern subscriptions if needed.A simple example of this is an RSS feed (“rich site summary” or “really simple syndication”). Say you’re interested in getting any news related to Game of Thrones. You can subscribe to an RSS feed that regularly checks with the servers in its list, pulls down any new information it finds on the show, and presents it to you.Of course, you can also use pub/sub in a business setting. Take the customer support ticketing example we used earlier to describe the routing/mediation pattern. The account manager can subscribe to any tickets involving a premier customer. Or a product manager may subscribe to topics related to his product roadmap.Pub/sub is easy to scale, secure and reliable. The architecture can support multiple back-ends that need access to the same data. Instead of 10 different integrations, you need only one.

As opposed to a point-to-point integration, where you have two applications communicating directly with each other, pub/sub has a message bus in the middle.A message bus combines a common data model and command set with a messaging infrastructure to allow different systems to communicate through a shared set of interfaces.Subscribers can choose the application they wish to use to receive the data, and they can choose the frequency with which the application checks the servers for new information.Many of the messaging buses in use also provide guaranteed delivery, so a subscriber doesn’t have to worry about missing an update.But there are drawbacks. Pub/sub is coupled to a protocol, meaning that subscribers must use the same protocol as the sender. And if the architecture wasn’t set up properly, management becomes difficult. For example, you would need to reconcile different queues with different permission settings.Furthermore, while pub/sub is much more scalable than point-to-point, the higher the volume of subscriptions, the more cumbersome it becomes to manage. And significant increases in message size can lead to performance degradation as those messages take up greater and greater storage space.That’s why we believe pub/sub is best for connecting applications within an organizations or with other applications where some performance degradation is acceptable.

DATA INTEGRATION PATTERNS IN ACTION

Source

Targets

Message Broker/Queue

11 eBook | Integration Patterns

3. APIs/Web Services

The rising popularity of cloud-based applications has spurred interest in application programming interfaces (APIs) to expose web services.An API is code that allows two software programs to communicate with each another. In the case of a web service, the API spells out the proper way for a developer to write a program requesting services from an operating system or other application via the web.The majority of APIs today use the Representational State Transfer (REST) protocol, an architectural style for distributed hypermedia systems — the web as we know it today. RESTful APIs allow users to connect and interact with cloud services, using less bandwidth than Simple Object Access Protocol (SOAP) technology. There’s no standard format in which the data must be processed, and once the API is created, everyone using it can get access to that data.This modern, lightweight way to unlock data abstracts the technical details, making it easy for the end user to consume digital services without worrying about the underlying technology.

For example, without the use of a RESTful API, connecting data between two cloud applications, such as PeopleSoft and SAP CRM, would require you to know the format of the fields, authentication, etc. In a RESTful API, that information is part of the metadata, making it simpler and easier to exchange information among cloud applications.The data can be exchanged without any application-level knowledge. Plus, authentication is very strong, providing more secure access than some other methods of communication. As the majority of web services are synchronous, RESTful APIs are best used for real-time, bidirectional data transfer. They are great for extending connectivity to other applications and third-party providers outside your organization. They are the preferred method of integration for many large SaaS providers.There aren’t many disadvantages to using RESTful APIs. Nevertheless, like any other connector, the higher the volume of data and message size, the more likely you are to confront performance issues.

DATA INTEGRATION PATTERNS IN ACTION

Client/User Source/TargetAPI Gateway ProcessingLogic

12 eBook | Integration Patterns

4. Extract, Transform, LoadMost organizations are familiar with extract, transform, load (ETL) — a proven method for transferring large volumes of data. It’s frequently used in data migration projects and for building data warehouses.ETL is really the only way to get large amounts of data in and out of legacy applications. It’s also a top option for data transfers with newer applications. ETL can handle vast quantities of structured or unstructured data.ETL extracts data from various sources, which is then cleansed, enriched, transformed and stored (in a data warehouse or data lake). Growing in popularity is a closely related alternative to ETL: extract, load and transform (ELT). In ELT, the extracted data is loaded into the data warehouse and then transformed.ELT pipelines often use cloud-based data warehouses such as Amazon Redshift and Google BigQuery because they’re highly efficient in performing transformations.These methods work nicely for moving large amounts of data that require complex rules and data transformations. The automation of processes makes maintenance and traceability much easier versus hand-coding. But because ETL and ELT are batch-driven, they aren’t a good fit for event-based or real-time integration. Plus, it can easily take months and a squad of database experts to build it and manage changing requirements.

DATA INTEGRATION PATTERNS IN ACTION

Source TargetTranslator

13 eBook | Integration Patterns

Section 3: Accelerate Your Business with a Unified Integration Platform

14 eBook | Integration Patterns

Dell Boomi accelerates your digital enterprise by making it faster and easier to unify data, systems, applications, processes and people. With proven connectivity to 1,000+ endpoints, Boomi’s integration cloud lets you quickly and easily connect any combination of applications — public or proprietary, on-premise or in the cloud — regardless of data format.Boomi can support any integration pattern or method — real-time, event-based, batch, ELT, asynchronous processes and more.

The Dell Boomi Advantage

Source/Target Source/Target

Client/User Source/Target

ProcessingLogic

APIGateway

MessageBroker/Queue

Translator

15 eBook | Integration Patterns

How Boomi Does It With the Boomi Unified Platform, you can easily address any integration scenario. Along with access to pre-built integration processes and reusable components, Boomi offers a vast library of technology and application connectors. These make it quick and easy for you to build simple to sophisticated integrations in our intuitive development environment.

Here’s how Boomi addresses the common integration methods:

Point-to-PointBoomi provides technology and application connectors to integrate and decouple applications, regardless of where they reside. Using Boomi also facilitates data mapping with Boomi Suggest. Boomi Suggest offers millions of indexed, successful mappings built by other users that automatically recommend data flows for new integrations.

Publish/SubscribeWith Boomi, you gain the benefit of format conversion, which a queuing mechanism cannot do on its own. Boomi can transform and send data to a publisher as well as convert data to read from an internal subscriber. Boomi provides an internal add-on queue for simple use cases. For more complex integrations, Boomi uses a JMS connector to tie to an enterprise queuing mechanism or other queuing technology.

APIs/Web ServicesIn an API-driven integration approach, Boomi acts as an enterprise service bus (ESB). We expose data via our API management tool and help applications unlock the target data. With Boomi, you can build modern applications that allow any kind of transformation. Boomi also supports all the security mechanisms for APIs.

ETL/ELTBoomi serves as an ETL process. After a certain data volume threshold, Boomi uses the ELT method, which has proven to be the most efficient way to manipulate huge amounts of data. For example, you can use our BigQuery connector (or an HTTP client or app connector) to extract and load the data, but you’ll need to transform it in BigQuery or with SQL once it’s in the destination system.

16 eBook | Integration Patterns

But integration is only the beginning. The Unified Boomi Platform offers organizations “economies of skill”— development and management efficiencies that help companies scale and grow faster.

Boomi Master Data Hub unifies data across systems, making it possible for analysts to model, match, synchronize, cleanse and enrich data spanning multiple domains. These comprehensive data management and governance capabilities ensure the highest quality of data while integrating with any combination of SaaS and on-premise applications.

And Boomi Flow provides a low-code development platform to create customer journeys and automate workflows — from simple to sophisticated. By abstracting code into visual models, Flow speeds development, helping roll out new applications and digital services faster. And it allows business stakeholders to offer input more effectively while IT maintains control over development and deployment.

Boomi API Management provides a unified and scalable cloud-based platform to centrally manage and enrich API interactions through their entire lifecycle. With Boomi, you can manage and control APIs, while rapidly creating and publishing any endpoint as an API on-premise or in the cloud.

If you need to manage a trading partner network, look no further than Boomi B2B Management. Rapidly onboard trading partners, support traditional EDI standards (as well as newer web services), and monitor all your partner interactions in real time.

Harness Economies of Skill to Unify Your Digital Ecosystem

17

To learn more about how the Boomi Unified Platform can help your digital enterprise move faster and with greater agility, please contact a Boomi integration expert today.

© 2019 Boomi Inc. Dell, Boomi, and Dell Boomi are trademarks of Dell Inc. or its subsidiaries. Other names or marks may be the trademarks of their respective owners.

Learn more about the Boomi integration platform

Boomi.com/platform

View all customer stories at Boomi.com/Customers

Connect on social

Modernize Integration Processes to Drive Business TransformationYour business needs a technology foundation that makes information, interactions and innovations flow faster inside and outside the organization. By modernizing integration processes, you can dramatically reduce the time and complexity required to connect systems and break down data and business silos.

There are several options to how you approach data and application integration. Each has pros and cons. But a modern, low-code integration platform makes it easy to adopt any of them. Boomi’s cloud-native, low-code platform allows you to start quickly, build efficiently and grow confidently.

Boomi helps unite everything in your digital ecosystem so you can achieve better outcomes, faster. Our intelligent, flexible, scalable platform accelerates business results by linking your data, systems, applications, processes and people.

By harnessing the power of the cloud to unify everything inside and outside of your business, Boomi creates a robust fabric of connectivity with the speed and agility to help lead your digital enterprise.