integrating sap and non-sap data for comprehensive...

www.barc.de/en

WHITE PAPER

Integrating SAP and non-SAP data for

comprehensive Business Intelligence

Business Application Research Center

2 Integrating SAP and non-SAP data

BARC – Business Application Research Center 2011

Authors

Timm Grosser

Senior Analyst

BARC

Carsten Bange

CEO

BARC

Business Application Research Center – BARC GmbH

Steinbachtal 2b

97082 Würzburg

Germany

+(49) 931 880651-0

[email protected]

This research note was conducted and written independently by BARC, an unbiased market analyst.

It can be distributed free of charge thanks to sponsorship from Composite Software.



Leveraging SAP data for Business Intelligence

During the recent economic crisis, many companies realized how important it is to quickly deliver

information on emerging business and market developments. Business Intelligence not only helps

companies get the insight they need on these and other important trends, it also delivers the neces-

sary analysis and forecasting tools to generate key information and manage business performance.

Yet, the fast fluctuations in external demand clearly showed that many Business Intelligence systems

can no longer meet today’s growing requirements for managing performance. In general, one ques-

tion has gained central importance: How can companies access data from their numerous operation-

al systems and integrate it with internal and external information from different transactional and

decision support systems for analysis?

In this context, accessing data from SAP sources is a prime focus of interest. SAP is the global market

leader for ERP systems and offers various operational and decision support systems. The options for

leveraging SAP data, however, vary from system to system. This difference is clearly evident between

its operational systems (e.g. ERP, CRM or SCM) and SAP BW, which was designed for decision sup-

port. Further, large enterprises often have multiple SAP versions and instances, adding further com-

plexity. As a result, companies need to seriously contemplate how they can integrate their SAP data.

– both directly from the source systems and consolidated for decision-making and analysis. This

study, therefore, will provide an overview of the different possibilities for accessing and integrating

SAP data as well as their advantages and disadvantages in light of the growing demands on Business

Intelligence in modern organizations.

Growing demands on Business Intelligence

The requirements for Business Intelligence systems have increased dramatically in recent years. Alt-

hough each company’s needs vary by industry, users, internal usage as well as the level of coopera-

tion with partners, certain requirements generally apply in most cases:

• Fast query responses: Users often become frustrated when they have to wait too long to re-

ceive queries from a Business Intelligence system. This factor alone can result in the failure of

a BI project. According to the BI Survey 9, the largest user survey on Business Intelligence,

poor query performance has been named one of the top three problems in BI projects in re-

cent years.

• Analyses across different systems: Companies often store their data in many different sys-

tems for specific departments and tasks. Multiple modules (e.g. ERP, CRM or SCM), multiple

versions, and multiple instances create even more complexity. This means that they have no

comprehensive, consistent view of their business activities. Gaining a single view of data is

becoming more and more important, for example, in order to identify opportunities.

• Integration of up-to-the-minute data: Business Intelligence is moving closer to operational

processes. In fact, it already supports them with analytics. In order to create a comprehen-

sive view across the entire enterprise, companies need to incorporate historical, highly ag-

gregated KPIs as well as detailed, operational data in their analyses. This requires a fast, easy

integration of the most current data.



• Open, easy-to-integrate architecture: Analyzing data from various internal and external

sources such as SAP systems and beyond is a key requirement for any BI application. In addi-

tion, the architecture should support multiple BI systems used for different tasks

• Validated pool of data: Poor data quality is a major problem in BI projects. The BI Survey 9

shows that poor data quality and its effects are becoming more and more important in com-

panies. Validated, tested data forms the foundation of any beneficial BI system.

• Scalability: New studies show that there are many other scenarios for using BI. Due to these

new possibilities, BI systems are growing in terms of users, data sources, data volumes as

well as the complexity of heterogeneous queries. As a result, BI systems must have a scalable

technological architecture.

• Agility: Companies must be able to implement new business, technical as well as functional

requirements in a fast, uncomplicated manner. To ensure that they can deliver important in-

formation to decision-makers on all levels of the organization, companies should be able to

quickly change existing objects as well as add new data sources, KPIs or dimensions in a time-

ly manner.

Access to SAP data varies by system

SAP ERP systems process critical business data from operational processes and, therefore, often form

the core of a company’s IT systems. Accordingly, SAP data is generally an important part if not the

sole base of data for reporting, analysis, planning and other Business Intelligence tasks. Nevertheless,

accessing and integrating SAP data – whether alone or with additional data sources – poses a major

challenge.

In general, there is a difference between accessing operational SAP data directly from the transac-

tional system or consolidated, analytical data from SAP’s data warehouse application, SAP BW.

Data in operational SAP applications

Many companies directly access the data in their operational SAP systems to monitor and report

these processes. SAP offers built-in functions for data analysis or reporting within its operational ERP,

CRM, PLM, SCM and SRM applications. Alternatively, users can utilize SAP BAPI and other SAP pro-

prietary interfaces to export data or to directly access SAP data for queries in external analytical tools

such as Microsoft Excel or a standard Business Intelligence tool.

Reporting directly from the operational system, however, is often insufficient. If a company has more

advanced requirements, there are clear drawbacks with regards to:

• Integrating external data

• Analyzing a combination of data from different modules

• Building new KPIs or analysis structures (especially hierarchies)

• Creating a history of data and allocating it into analytical structures

• Delivering sophisticated options for formatting or visualizing data (e.g. in management dash-

boards)

• Supporting BI tasks that go beyond operational reporting such as advanced or flexible data

analysis, forecasting and planning



The BI tools that are available in operational SAP systems are traditionally limited to simple reporting

tasks. If companies have requirements that go beyond that, they almost immediately need to use

complementary SAP and non-SAP BI solutions to analyze, visualize and distribute SAP data in a suita-

ble manner. For BusinessObjects users we expect these possibilities will improve in the mid to long

term through the stronger integration of BusinessObjects tools with operational SAP systems.

The drawbacks described above arise from the general character of operational systems. For starters,

these systems can only process their own data for analysis. Since there is generally no possibility to

integrate external data, creating a complete analysis across different operational SAP modules and

systems can be very difficult. Many companies, too, have implemented their various ERP installations

in many different ways and the data models of these applications – for example, SAP ERP and CRM –

can differ as well. In addition to requiring unique structures and KPIs, Business Intelligence generally

needs to access data for longer periods of time as well as transform it into analytical data models.

This functionality, however, is not available or highly limited in operational systems.

These operational systems limitations led to the development of SAP BW, SAP’s data warehousing

solution. According to SAP, 10,000 customers – a third of SAP’s estimated 25,000-30,000 ERP cus-

tomers – use this system.

Data in SAP BW

SAP Business Warehouse (SAP BW) provides access to information formatted for analysis in the form

of a data warehouse that runs complementary to the SAP operational systems. Its specialized access

mechanisms for extracting data from operational SAP systems as well as transferring it into data

structures that are optimized for analysis generate a consolidated warehouse for business decision

making.

As a classical data warehouse, SAP BW supplies this information through interfaces for further pro-

cessing. Yet, accessing data in SAP BW is subject to a few limitations:

• BI Consumer Services (BI-CS), the most powerful interface for user tools, only works in com-

bination with SAP BusinessObjects. In addition, the write-back interface for data entry and

planning applications is generally not open to other applications.

• OLAP BAPI, MDX, XMLA, the open front-end interfaces of SAP BW, are known to cause vari-

ous usage problems including functional limitations, performance issues and restrictions in

the amount of readable data.

• If a company wants to use a tool which accesses data that is exported from SAP BW and not

from direct queries to that data warehouse, it will have to purchase a data export interface

which has a list price of € 250,000 – a major licensing hurdle.

In addition to these limitations with regards to access, other attributes of SAP BW often provoke

companies to maintain separate tools and additional data warehouses and marts for Business Intelli-

gence.



Diagram 1: This parallel data warehouse approach uses SAP BW and a separate analytical database which combines and

integrates SAP data with information from heterogeneous source systems for analysis.

There are many different reasons for this development:

• There are limited options for integrating data from non-SAP systems in SAP BW because

connecting standard applications to the SAP system or extracting changed data from external

sources requires a significant amount of specialized programming or third-party data integra-

tion tools. Typical SAP BW applications generally use data from operational SAP systems –

and little data from other sources.

• End users often experience poor query times if they use SAP BW without BW Accelerator.

Optimizing query speed requires a significant amount of administrative work and is rather

difficult in environments with heterogeneous query profiles. Purchasing BW Accelerator to

speed up queries is not an option for many companies due to the high investment costs.

• There are few possibilities for exchanging metadata between SAP BW and the semantic view

of BI user tools.

• In SAP BW, it is very difficult to enrich historical reports with up-to-the-minute data.

• Adding data sources or changing existing models is relatively complex in SAP BW.

• Analysts must have MDX expertise, whereas SQL domain knowledge is more widespread.

• Some Business Intelligence tools have certain requirements on the database technology.

Especially in reporting, planning and data mining, users need to create real-time calculations,

manipulate data or simply access information using standard SQL queries. Sometimes, the

tools and applications for these tasks only work in combination with certain databases, which

then are used in combination with SAP BW.

As a result, many companies have a growing number of systems that contain SAP data in various

formats and need to create a common strategy to access this data as well as other relevant infor-

mation to make decisions. The technical solution to support this type of strategy is an integrative

layer that enables users to get a complete view of business information from SAP systems and other

data sources.



Solution architectures for integrating SAP and non-SAP

data for enterprisewide Business Intelligence

SAP data is located in SAP BW as well as many different operational systems from SAP. In addition,

users often need to access information from other operational systems, other data warehouses and

data marts as well as additional internal and external databases for comprehensive Business Intelli-

gence. Increasingly, companies are seeking a unified information architecture that integrates all of

these data sources and systems completely and consistently across their enterprise. To do this, com-

panies can generally apply one of three approaches. They can:

• Physically integrate the data in an all-inclusive, enterprisewide data warehouse,

• Create an enterprisewide, shared data layer that allows virtual integration without physically

moving the data from the source systems, or

• Use a hybrid combination of physical and virtual that maximizes the benefits and minimizes

the limitations of each approach.

Enterprise data warehouse via physical data consolidation

The principles and business value of an all-inclusive, enterprisewide data warehouse for enabling

business decision-making are well understood and illustrated in diagram 1. The advantages of this

approach are:

• Consolidating and reorganizing large volumes of data from heterogeneous systems

• Building an analytical view by creating multidimensional models of relevant objects for the

decision-making process

• Saving data and analytical structures over a longer period of time (5-10 years)

• Decoupling analysis from the operational systems

• Supplying data for many different analytical scenarios with optimal performance

• Creating complex calculations within the database as well as for data mining applications

The disadvantages of this approach are:

• Creating a second system requires additional resources and software licenses

• Companies need an Open Hub service license to integrate SAP BW data and build a data

warehouse or analyze the data using third-party tools

• Integrating data from standard applications (e.g. SAP) requires special interfaces and func-

tions

The key criteria for evaluating enterprise data warehouse and supporting data integration infrastruc-

ture are also well known. Companies can use SAP BW as the all-inclusive enterprise data warehouse.

Due to the restrictions described above, however, most companies use only SAP BW as a data mart

for executing specific tasks or evaluating SAP data in combination with other data warehouse and

reporting solutions.



Enterprise data layer via data virtualization

An enterprisewide data layer provides an alternative to the traditional enterprise data warehouse. A

virtual layer provides a simple, flexible way to integrate and provide access to data from many differ-

ent systems. Enterprise-scale data virtualization platforms have emerged from earlier Data Federa-

tion or Enterprise Information Integration (EII) tools to provide the functionality required to create

such a layer. These offerings include data discovery, modeling and design capabilities to prebuild the

integrated views of the data required from across the entire range of source systems. These views

can be traditional database views, data services or both as appropriate. When a user’s BI solution

requests information, the data virtualization server accesses, transforms, federates and presents the

data in the defined target structure at the moment a query is made.

Diagram 2: In the integration layer data from heterogeneous sources (including SAP systems) is combined and provided

for Business Intelligence.

The advantages of this approach are:

• Fast, easy integrations of new or temporary data sources

• Synchronized access to decision support and operational systems

• High flexibility for making changes since they don’t need to be made to physical structures

and data

• No need for transferring data since it doesn’t need to be replicated from the operational or

decision support systems (or their operational data store equivalents)

• Simple integration of up-to-the-minute data and new data sources

This approach, however, also has its disadvantages:

• Despite optimization and caching measures, companies still need to check if performance

meets service level requirements.



• Ensuring data quality can be a challenge if complex, multi-step data cleansing is required. As

a result, companies must ensure that their data is valid before the integration process takes

place.

• Large analytic queries whose datasets require significant re-dimensioning or time-series

summations, etc. are not appropriate.

Building an integration infrastructure with data virtualization capabilities is a new approach for many

companies. To evaluate vendors and products for this innovative technology we recommend includ-

ing these criteria:

• Ease-of-use and productivity in modeling and development

• Performance & scalability

• Connectivity to a wide range of data formats, databases and applications, especially certified

SAP ERP and BW specific data access functionality

• Integration with security concepts, e.g. SAP security and implementation of security concepts

in the virtual data integration layer.

• Mechanism to ensure reliable, enterprise-scale operation and maintenance

Hybrid Architecture combining enterprise data warehouse and enterprise data layer

As a common information architecture that integrates SAP operational, SAP BW, and various non SAP

sources for comprehensive Business Intelligence, both the enterprise data warehouse and enter-

prisewide data layer have advantages and limitations. However, by effectively mixing and matching

these approaches, companies can continue to benefit from the advantages, while mitigating many of

the disadvantages.

Diagram 3: In the integration layer data is provided by combining the virtual and the enterprise data warehouse ap-

proach for Business Intelligence.



The “mix and match” approach obviously means adding a system to the existing architecture that

needs upfront investment and resources to run. But the advantages are often compelling in situa-

tions where these requirements exist today:

• High flexibility in implementing new business requirements using an easily adaptable data

model

• Fast integration of new data sources, up-to-the-minute or missed data to extend the data

available in the data warehouse

• Combine operational and analytical master data from master data management, data ware-

house and operational systems

• Create a common view on data stored in disparate data marts or warehouses

• Adding to the capabilities of existing data warehouses without having to fundamentally

change the systems

Summary and recommendations

The demands on Business Intelligence tools have increased dramatically in recent years. One increas-

ingly important requirement is the creation of complete views of business information by integrating

data from different systems across and around an enterprise. In this regard a prime area of interest is

data from the operational and decision support systems of SAP. Although SAP systems offer func-

tions and interfaces for Business Intelligence, they have their shortfalls. This results in multiple cus-

tomized SAP systems as well as a multitude of systems containing SAP data. In order to create a ho-

listic view on relevant information for making decisions, companies need to consolidate data from

various systems into unified, enterprisewide integration architecture. Currently, they can choose one

of three different architecture approaches: an all-inclusive, enterprisewide data warehouse, an en-

terprisewide virtual data layer, or a hybrid combination of the two.

In all three solution approaches, the open interfaces and support for standards increase the interop-

erability with other systems and create new potential (e.g. by integrating specialized BI tools) that

SAP tools alone could not offer. This way, companies can also secure their SAP investments because

they react to changing requirements without having to completely rebuild or replace their current

systems.

The following aspects are often cited as advantages of physically integrating information in an enter-

prisewide data warehouse:

• If query speed is a very important factor, different database technologies – including multi-

dimensional and column-based relational databases or innovations such as in-memory data

storage – provide a broader range of options using physical data integration.

• A physical integration into an all-inclusive data warehouse is more suitable than a virtual in-

tegration when it comes to creating complex calculations and analyses, transferring large

volumes of data and mapping histories or deeper dimensions.

• If a company needs to improve the data quality between its operational systems and queries,

they can implement data quality management tools during the transformation process be-

tween source systems and the data warehouse.



The enterprise wide data layer, in contrast, has advantages in other scenarios:

• By building and deploying virtualized views and data services, companies can meet their

growing requirements for agile BI. Enterprises can generally change existing view and ser-

vices, as well as integrate different data sources on a regular basis more quickly using this

approach.

• Data virtualization allows companies to combine many different types of data sources more

flexibly. It also allows for running test scenarios on the fly and applying other analytical

methods faster.

• The virtual approach is particularly useful for integrating up-to-the-minute data because

companies can directly access source data without waiting for ETL batch updates.

Thus, for companies with a wide range of requirements, a hybrid approach combining the two ap-

proaches shows multiple advantages:

• The hybrid approach allows the computing power of parallel data warehouse systems and

analytical databases to be used for complex analytical queries and processing high volumes

of data. Using the virtual approach at the same time increases the system flexibility and agili-

ty, for example by making it easier to integrate data sources or to implement new business

requirements.

• The hybrid approach offers an infrastructure that makes it easy to combine historical and val-

idated data of a data warehouse with new or up-to-the-minute data. This allows comprehen-

sive BI analysis and meets growing business requirements.

• The hybrid approach can facilitate the BI roll-out. Even when covering a wide range of busi-

ness needs preliminary results can be returned very quickly. Fast implementation is a key fac-

tor for project success and BI marketing within companies to win over a broader range of

business users and sponsors.

We therefore recommend that all companies identify their requirements and pain points in the situa-

tions described in this paper to evaluate the new possibilities of a virtual data integration infrastruc-

ture – either as an add-on or an alternative to existing systems for integrating data from SAP systems

and other data sources.



BARC stands for neutrality, competence and quality

Neutrality

The Business Application Research Center (BARC) was founded in 1994 as a spin-off from the Univer-

sity of Wurzburg’s Chair for Information Science. BARC is an independent institute and treats all

software vendors with the same objectivity. It does not accept fees for coverage in software evalua-

tions or commission for software recommendations. BARC also does not offer software implementa-

tion services to prevent internal conflicts of interest.

Competencies

BARC employees have assessed Business Intelligence and Data Management products and consulted

companies in this field since 1994. BARC analysts have vast market, product and implementation

insight. Their comprehensive, yet detailed knowledge of the latest market developments and the

offerings of all relevant software vendors is backed by years of dedicated market research and

hands-on product evaluations.

Quality

BARC consulting projects are highly efficient and offer the highest level of security for software selec-

tions. BARC reports provide competent market overviews for all applied areas of information man-

agement. BARC conferences and seminars offer a concentrated overview of key players in a specific

segment of the software market.

integrating sap and non-sap data for comprehensive...

Documents