data warehouse data flow in sap bw - guidance consulting

SAP BW CLASS DATA WAREHOUSE –

DATA FLOW IN SAP BW

Data Warehouse – Data Flow in SAP BW Guidance Consulting

Copyright© 2007 by Guidance Consulting, Houston, TX Disclaimer The information in this publication is the copyrighted work of Guidance Consulting and its owner Touseefuddin Syed and is owned by Guidance Consulting. While there is no official print of this document, this book and the information herein are the sole property of Guidance Consulting. This is covered under the US laws of copyright. Hence, no part of this book in part or in its entirety form can be distributed or circulated in any form of media without the prior written consent/permission of approval from owners of Guidance Consulting. Any person quoting out of this booklet should properly document their work with a reference to this document with the consent of the owner/publisher of this document. Any person, group or organization that will breach copyright law knowingly or unknowingly will be prosecuted to the full extent of the law of this country. NO WARRANTY: This documentation is delivered as is, and Guidance Consulting makes no warranty as to its accuracy or use. Any use of this documentation is at the risk of the user. Although we make every good faith effort to ensure accuracy, this document may include technical or other inaccuracies or typographical errors. Guidance consulting, reserves the right to make changes without prior notice. NO AFFILIATION: Guidance Consulting, and this publication are not affiliated with or endorsed by SAP® AG, SAP® AG software referenced on this site is furnished under license agreements between SAP® AG and its customers and can be used only within the terms of such agreements. SAP® AG and mySAP® are registered trademarks of SAP AG. This document also uses references to so many products hence all other product names used herein are trademarks or registered trademarks of their respective owners such as Microsoft®, Oracle®, IBM®, etc.

Copyright © Guidance Consulting Instructor: Touseefuddin Syed

2


Data Warehouse Modeling - Dataflow Introduction To get to the definition of Data Warehouse, it would be inappropriate not to consider the definition given by William Inmon (considered to be the father of Data Warehousing). He defined a data warehouse as follows: “1) Subject-oriented: Meaning that the data in the database is organized so that all the data elements relating to the same real-world event or object are linked together. 2) Time-variant: Meaning that the changes to the data in the database are tracked and recorded so that reports can be produced showing changes over time. 3) Non-volatile: Meaning that data in the database is never over-written or deleted, but retained for future reporting. 4) Integrated: Meaning that the database contains data from most or all of an organization's operational applications, and that this data is made consistent.” Simply put together, information/data in the most important and valuable asset a company has and how they can utilize this data, forms the basis of a successful company. For example, an organization would use the information that's stored in its data warehouse to find out what day of the 1st week they sold the most cell phones of a particular model in February 2007, or how employee sick leave the week before holiday season in December differed between Houston and Chicago from 2005-2007. OLTP systems can perform reporting tasks but has its limitation. As we discussed earlier, OLAP Systems has been adopted by many organization to oversee the complications and overcome the limitations produced by OLTP systems. For data from any source (SAP or non-SAP sources) and of any age (historic or current), Data warehousing in Business Intelligence (BI) allows: 1) Integration (data acquisition from source systems) 2) Transformations 3) Consolidation 4) Clean up 5) Storage 6) Retrieval for analysis and interpretation Data warehousing in BI allows you to directly access source data as well as physically storing data in BI. In the foregoing pages, we will discuss the process of dataflow in the SAP BW environment.


3


Dataflow in BW Definition It shows the flow of data from the source system to the InfoProvider where it gets readied for use in the reports. Simply put together “the data flow is the path that data have to follow (among several stages and provider) in order to have all the info available for final reporting purpose”. - Very helpful to see the related structures in bundles - You can also see the technical names for them on the display screen (you will have to turn on technical

name switch ) Sample dataflow screen from RSA1 (Administrator Workbench: Modeling) Click on InfoProvider on the left panel Then look for the particular cube you want to see the dataflow for Right click on it and choose “Show Data Flow”, you will get to this screen FIG 01.

FIG 01: Dataflow


4


Dataflow in Detail We will go in details in each of these objects starting from the bottom. Source Systems Systems that provide BI with data are described as source systems FIG 02. Different source systems a) SAP systems (such as R/3, SAP CRM, SAP APO and SAP SEM) b) BI systems (Hyperion, …) c) Flat files (MS Excel, …) d) Database management systems into which data is loaded from a database supported by SAP using DB

Connect, without using an external extraction program (IBM, Oracle …) e) Relational or multidimensional sources that are connected to BI using UD Connect (IBM, DBL …) f) XML Data (through SOAP protocol): Web Services that transfer data to BI by means of a push. g) Non-SAP systems for which data and metadata is transferred using staging BAPIs (legacy applications

Oracle Financial, Ascential DataStage and Informatica PowerCenter)

FIG 02: Source Systems Creating Source System in BW

1) Go to RSA1 and click on , you will come to SOURCESYSTEMTREE screen.

2) Right click on and select , you will come to this screen,

FIG 03: Select Source Systems Type 3) Select the most relevant source system. In our case we select first option


5


and click ., FIG 04 shows this screen,

FIG 04: Create SAP Source System 4) This information should be provided to you by basis team, team lead and/or project manager.

a) “Available destination” should have this kind of entries b) Select the relevant clients and logon as a system administrator. i) First make sure that you have the authority to create users and RFC destinations.

ii) The RFC destinations for BI and the background users are thus created automatically in the source system. If the RFC destination already exists in the source system, check its accuracy iii) You can test the RFC destination using the function Test → Links and Test → Authorizations

* RFC stands for Remote Function Calls is the SAP interface protocol, used for extraction of data from SAP R/3. (This is related to advance topics that will also involve some basis knowledge, that’s why we will not discuss this anymore in this class).

c) The ALE settings, which are needed for the communication between a BI System and an SAP System, are created in the background with the use of the created destinations. These settings are made in BI as well as in the source system. The BI settings for the new connection are created in BI. * ALE stands for Application Link Enabling is the proprietary SAP technology for communication between Source System and BI Systems.

d) If the new SAP source system has been created, metadata is requested automatically from the source system. The metadata for DataSources is also replicated to BI in the D version.


6


Datasource SAP describes Datasource as “Data that logically belongs together is stored in the source system in the form of DataSources”. So to say that DataSources are used for extracting data from a source system and for transferring data into BI. 1) DataSources are used to extract and stage data from source systems. The DataSource can be used for all

objects that contain data i.e. Master data (Attribute, Text, and Hierarchy) and Transactional data. 2) Or for that matter the data is loaded into BI from any source in the DataSource structure. 3) The structure for transferring data from a DataSource to SAP BW is called transfer structure (we will

discuss this later in the class under the same topic) 4) The DataSources subdivide the data that is provided by a source system into self-contained business areas. 5) An InfoPackage is used for this purpose. You determine the target into which data from the DataSource is to

be updated during the transformation. You also assign DataSource fields to target object InfoObjects in BI. 6) SAP R/3 systems provides a number of SAP Business Content DataSources that can be used immediately. We will discuss DataSources in detail in the following classes when we actually create a DataSource for the purpose of this class. PSA (Persistent Staging Area) PSA (Persistent Staging Area) is an inside storage place where the data is stored in an unchanged form from the source systems. - InfoPackage’s load the data from the source into the PSA - Is stored in Transfer Structure format as a transparent PSA table is created for every DataSource that is

activated. - In the relational database tables as in BW systems, the PSA tables each have the same structure as their

respective DataSource. - Data is not summarized/transformed and the duration of the data storage in the PSA is medium-term.

Deleting PSA Data There will be situations when you will have to delete data in the PSA. One possible reason could be that it has outgrown its size. Follow these simple steps to do so:

1) From RSA1 (Administrative Workbench - Modeling, click on

2) Right mouse click on this PSA node on this screen and choose from the menu. You can do this from individual nodes from where you want to delete PSA data as well.


7


FIG 05: Deleting PSA data 3) You will come to this screen

FIG 06: Delete PSA Data a) From this screen, you can select no. of days the data older that needs to be deleted. b) Here you can select the date you want the data to be deleted before certain date (mm/dd/yyyy).

c) Click on to get to this screen


8


FIG 07: PSA Start Time

d) You can start the deletion , schedule , after completion of a certain job

, after an event is triggered , and at and also we make it as a

by checking this box.

e) Click to come back to “Delete PSA” window and then click start the deletion.

You can achieve this by Process Chain using the Process type “Deletion of requests from PSA". Process chains are available from BW 3.0, which we will discuss later in the class under the same topic.

Transfer Rules Definition and Purpose

• The transfer rules transform and clean up the data from the PSA. • Transfer rules determines how the mapping is done from the Datasource fields to the InfoSource

InfoObjects. • Transfer rules are dependent on the source system data and reflect the same • The no. of transfer rules would be equal to the no. of source system for a data target • Transfer rules is mainly for data cleansing and data formatting after the data arrives in the

activation queue • Here the transfer structure and the communication structure are maintained • You use the transfer rules to determine how you want the transfer structure fields to be assigned

to the communication structure InfoObjects. • You can arrange for a 1:1 assignment. You can also fill InfoObjects using routines, formulas, or

constants.


9


• When an InfoPackage is created, a source system, a DataSource and an InfoSource are needed to be selected. So, essentially, we are creating transfer rules between the DataSource and the communication structure of the InfoSource.

InfoPackage’s are meant for scheduling the load and they send a request Idoc to the source system to fetch the data. You would create InfoPackage’s for each of the source system which is based on transfer rules.

Creating Transfer Rules We have already created DataSource in the earlier scenario, now in order to create transfer rules, we will need to have an InfoSource created (it will be explained in the next section). So from the AWB (RSA1),

we click on under modeling. Then on the right hand side we double click on any

InfoSource (e.g. ZFISL related) or right mouse click on the InfoSource and choose . It is possible you will get this screen,

FIG 08: Error Object Editing window

Keep hitting , and it will take you to this screen as given in FIG 09,

FIG 09: InfoSource Display


10


Click and Expand , you will come to this screen, see FIG 10.

FIG 10: Transfer Rules 1) This line shows the source where the data

originated from. Click on the drop down arrow and choose a different source

system , if it is different. 2) This line shows the DataSource from the

data is coming from.

3) Transfer method shows once the DataSource was replicated, the data was transferred to PSA.

4) Circle 4 is where you will click and put your cursor (e.g. ) there in the “Assign InfoObject

Field” and then you click on circle 5 (e.g. ) and then circle 6 (in normal scenario, this icon will be enabled and not disabled as I am showing in this example). Now they will be connected and then click on circle 7 .


11


5) You can click on circle 8 to specify “Transfer Rules”. You will come to this screen “Edit Transfer Rules”. See FIG 11.

FIG 11: Edit Transfer Rules 6) Here you can specify whether this is a “InfoObject” or “Constant”. You can also write a ABAB

routine and customize it, for e.g. changing the currency type. You can also specify a Formula here and do some calculations.

7) If you click on circle 9, this will allow you to search for specific objects in the list. Updates Rules Concept and purpose - Generally, the update rules are only used here for one-to-one transfer into the ODS object - Update rules contain the business logic and its nature depends on how we want the data target i.e.

InfoCube to be. - Update rules specify how the data (key figures, time characteristics, characteristics) is updated into

the InfoProvider from the communication structure. - Update rules are Data target dependant - In Update rules you would write the business rules (called routines which in done in ABAB) for

your data target for e.g. Currency translations are possible in update rules and some other more complex computations.

- If u would want to split a single record into multiple values, return table is an ideal solution in update rules (we will not discuss return table here as it is an advanced topic).

For e.g.: We have sales revenue value for a particular customer. Now if it is needed to be split into percentages based on an additional character say segment (and this not a standard field in R/3), the single record with one value per customer can be split into these percentages using the return table.

How are Update Rules and InfoPackage’s related? In the InfoPackage, as data targets you can select those InfoProviders that have an Update Rule connected to the InfoSource in which you are creating the InfoPackage.


12


Creating Update Rule

1) Update Rules are technically between InfoSource-ODS and between ODS-InfoCube, so to say between InfoSource and InfoProviders.

2) From AWB-Modeling (T/C: RSA1), click on in the left window pane and from the right window, choose the InfoProvider we want to create Updates Rules on.

FIG 11: Choosing InfoProvider to create Update Rules

3) Right mouse click on the InfoProvider (here it is InfoCube FIAR: Line Items with tech name:

ZFIAR_C03) and choose . 4) You will come to this screen, click on to choose the InfoSource.

FIG 12: Create Update Rules This will open the InfoSource selection screen


13


5) You will come to this screen,

FIG 13: InfoSource selection

a) This is the InfoSource selection screen, it will also show how many entries it has.

Even the ones we install from Business Content will be seen here. b) Scroll down to look for the one you are looking for. You can also and then type

in the name of the InfoSource (full or in part) and search for it. c) Highlight the one you want to select and

d) Click to come back to the Update Rules screen. InfoSource - An InfoSource is a set of all logically associated information. - InfoSource can contain Transaction Data (which is stored in InfoCube) and Master Data (which has

Attributes, Texts and Hierarchies and these data is stored in different tables) - InfoSource describes all the transaction available for a business transaction or a type of business

transaction. - As discussed before here, Transfer structures support the transfer of data in a DataSource between a

Source System and the associated SAP BW System. The transfer structure transports the DataSource data from a Source System to an SAP BW System and passes it on to the InfoSource using transfer rules.

- The Communication structure is generated from InfoSource and is independent of Source System. It is filled from the transfer structure in accordance with the transfer rules.

- The Communication structure contains all the fields in an InfoSource.


14


- Each InfoCube can have one set of Update Rules that determines how the data in the InfoSource is stored in the InfoCube

InfoPackage Info-packages trigger the data load into BW system from the source system. An info-package is created for a specific data source and info source combination. 1) When you trigger the info-package it sends a request to the source system to start the extraction process and this corresponds to the step "Data Request Received" in monitor under details tab. It is the first step. 2) Then the selections specified in the info-package are transferred to the source system to select the requested data from BW system. Then it is sent back to BW system. 3) Once it reach BW, it gets into the PSA and then it gets updated into the data target (depends on what settings are made in IP). No rule (Transfer rules / update rules) comes into picture when data is stored in PSA. Only the transfer structure is used to map the BW info-objects with the source system fields. 4) After data is stored into PSA, the update into the data targets begins. First the transfer rules are applied -> data then flows through the communication structure -> update rules are applied and finally the records gets updated into the data target. The extraction flow is: - Source System (DS) -> PSA -> Transfer Rules -> Communication Structure -> Update Rules -> Data target. Info-package actually triggers the whole data flow in BW system. References 1) William H. Inmon, Richard D. Hackathorn: Using the Data Warehouse, John Wiley & Son's 2) Few extracts taken from wikipedia.com and the site URL is: http://en.wikipedia.org/wiki/Data_warehouse 3) Data modeling book referred, by SAP Press. 4) Class notes referred from Vinay Karna of RD Data Solutions. 5) Several SDN and help.sap.com links that helped me create this document. 6) http://help.sap.com/saphelp_nw04/helpdata/en/e3/e60138fede083de10000009b38f8cf/frameset.htm 7) http://help.sap.com/saphelp_nw04s/helpdata/en/fc/1251421705be30e10000000a155106/frameset.htm 8) http://help.sap.com/saphelp_nw2004s/helpdata/en/a8/6b023b6069d22ee10000000a11402f/frameset.htm 9) http://help.sap.com/saphelp_nw2004s/helpdata/en/e9/6bf2d90e533f409ee56d3f586c325a/frameset.htm 10) https://www.sdn.sap.com/irj/sdn/thread?forumID=131&threadID=206135&messageID=2286851#2286851 11) https://forums.sdn.sap.com/thread.jspa?forumID=131&threadID=206135&messageID=2286851#2286851 12) https://www.sdn.sap.com/irj/sdn/thread?forumID=131&threadID=174856&messageID=1959651 13) Extensively referred to Vinay Karna’s notes.


15

data warehouse data flow in sap bw - guidance consulting

Documents