dmm300 – mixed scenarios for sap hana data warehousing and bw: overview and experiences

Public

DMM300 – Mixed Scenarios for SAP HANA Data Warehousing: Overview and Experiences

© 2016 SAP SE or an SAP affiliate company. All rights reserved. 2Public

Speakers

Bangalore, October 5 - 7

Edwin Dayanandh

Las Vegas, Sept 19 - 23

Sebastian Baumgärtner

Barcelona, Nov 8 - 10

Oliver Schall


Disclaimer

The information in this presentation is confidential and proprietary to SAP and may not be disclosed without the permission of SAP. Except for your obligation to protect confidential information, this presentation is not subject to your license agreement or any other service or subscription agreement with SAP. SAP has no obligation to pursue any course of business outlined in this presentation or any related document, or to develop or release any functionality mentioned therein.

This presentation, or any related document and SAP's strategy and possible future developments, products and or platforms directions and functionality are all subject to change and may be changed by SAP at any time for any reason without notice. The information in this presentation is not a commitment, promise or legal obligation to deliver any material, code or functionality. This presentation is provided without a warranty of any kind, either express or implied, including but not limited to, the implied warranties of merchantability, fitness for a particular purpose, or non-infringement. This presentation is for informational purposes and may not be incorporated into a contract. SAP assumes no responsibility for errors or omissions in this presentation, except if such damages were caused by SAP’s intentional or gross negligence.

All forward-looking statements are subject to various risks and uncertainties that could cause actual results to differ materially from expectations. Readers are cautioned not to place undue reliance on these forward-looking statements, which speak only as of their dates, and they should not be relied upon in making purchasing decisions.


Agenda

Introduction / Motivation for Mixed Case Scenarios

Overview of Mixed Scenario Architectures

Experiences with: BW on SAP HANA Transformations

Generated SAP HANA Views

Integrating SAP HANA Views

Public

Introduction / Motivation


SAP HANA

Mixed Architecture with SAP BW and SAP HANADefinition

SAP’s best practice for modern data warehousing

A Mixed Architecture consists of an overall data model that is implemented at the same time by BW and native SAP HANA tools It combines processes, data and metadata of BW and SAP

HANA native (= best-of-both-worlds to gain additional insight and flexibility)

SAP HANA and SAP BW deliver integrated tools to manage mixed scenarios end-to-end including modeling, transport mechanisms and consumption interfaces.

SAP HANA Database as central runtime for reporting and data warehouse processes offering best in class performance and scalability for large data warehouses

SAP BW

Sources

BI Clients

SAP HANA Modeling

BW Modeling

<

Mixed Architecture

SAP HANA Data Warehousing: Models for SAP BW and SQL DW on SAP HANA

DMM302 (L1)i



Mixed Case Scenarios

In addition to “classic” BW scenarios, pure native modeling capabilities are now available as well as BW powered by SAP HANA (BW optimized with SAP HANA features, e.g. BW on SAP HANA Transformations)

More possibilities / larger flexibility

New know-how / guidelines / best practices required

This session aims to provide insight / experiences how to approach modeling and shows different options



Mixed Case Scenarios

Application requirements and the existing landscape are drivers for a new architectureand its focus area

We will introduce selected options in an overview before showing best practices for detailed implementation aspects

Public

Architecture Overview



Staging Capabilities

SAP BW provides SAP HANA enabled Transformations with optimizations for standard functionality as well as the ability to use SAP HANA Expert Script (“SQL Script Exit”)

SAP HANA

SAP BW

Re

po

sit

ory

Sources

Data Warehousing

SAP HANA Views

SQLScript

Data Acquisition



Generated SAP HANA Views for BW Info Providers

Neglects reporting layer and does use BWs staging capabilities

Often driven by client decision which does not have a tight integration with BW backend (BW query, BICS)SAP HANA

SAP BW

Re

po

sit

ory

Sources

BI Clients

Data Warehousing

SAP HANA Views

gen.SAP

HANA Views

SAP HANA Views

Data Acquisition



Generated SAP HANA Views for BW Queries

Alternative to previous scenario with usage of analytic engine

Re-use of certain query properties possible (e.g. restricted / calculated keyfigures) for BI clients with shallow BW integrationSAP HANA

SAP BW

Re

po

sit

ory

Sources

Data Warehousing

gen. SAP

HANA Views

Data Acquisition

Analytic EngineBW Query BW Query

gen.SAP

HANA Views

Wo

rks

pa

ce

s

BI Clients



Integration of modeled SAP HANA Views from agile scenarios

Replicated and processed data from a separate schema is integrated into BW

SAP HANA view is consumed in a Composite Provider / Open ODS View and mixed with BW dataSAP HANA

SAP BW

Re

po

sit

ory

Sources

BI Clients

Data Warehousing

Data Acquisition

Analytic Engine

BW Query

Composite Provider

SAP HANA Views



Integration of modeled SAP HANA Views from agile scenarios

Data management completely native

Integration into BW to use feature-set of analytic engine

SAP HANA

SAP BW

Re

po

sit

ory

Sources

BI Clients

Analytic Engine

BW Query

Composite Provider

SAP HANA Views

Public

BW on SAP HANA Transformations

SAP HANA

SAP BW

Re

po

sit

ory

Sources

Data Warehousing

SAP HANA Views

SQLScript

Data Acquisition



Why to use Transformations with SAP HANA Execution?

SAP HANA enabled transformations avoid data roundtrips between application server and database

Better parallelization inside the SAP HANA database

Better performance / more flexibility (adaptation of SAP HANA features)


How do Transformations with SAP HANA Execution work?

After the design of the transformation capability which processing mode is possible

Similar to generated program for ABAP the generated SAP HANA Transformation can be displayed




SAP HANA Transformation(SAP HANA Analysis Process)

CalculationScenario



Prerequisites for Transformations with SAP HANA ExecutionSupported targets: Standard + Write-optimized DSO, Advanced DSO (ADSO), SPOs based on DSOs, OpenHub destinations for DB tables or 3 rd party tools

Queries as Info Providers are not supported as a source

ABAP routines are not supported (field routine, characteristic routine*, start routine, end routine, expert routine)

Rule groups are not supported

Read from DSO entire key must be specified

Near-line connections are not supported

*0SOURSYSTEM and 0LOGSYS are supported



The DTP combines Transformations between two Info Providers with persistence.

SAP HANA Expert Script0BW_OPER_PROC

Standard0BW_PROJECTION

Standard0BW_PROJECTION

HAPTR_00O2TMRZFVSBOQ6FLWQERAEAH

HAPTR_00O2TMRZFVSBOQ6FLWQERAEAH2

HAPTR_00O2TMRZFVSBOQ6FLWQERAEAH1

DTP_00O2TMRZFVSBOQ6FLWQERAEAH



The calculation scenarios for the related transformations are stacked.



During the SAP HANA Execution of a DTP (availability can be checked) an package-wise “insert as select” statement is generated



Prerequisites for SAP HANA Execution Availability of a DTPNo semantic grouping

Data target is DSO “subsequent processing without master data” is selected

Data target is Open Hub data target is database table or 3rd party tool

Data source is DSO Delta Init from Active Table (with Archive) not set

Data source is BW DataSource Extraction is performed from PSA table



Novel BW 7.50 SP04 featuresIn data flows with multiple transformations SAP HANA Expert Script and ABAP Transformations can be mixed (SAP HANA Expert Script must be first transformations)

Data Target with SIDs are also supported (e.g. Info Cube, Info Object)

Start-, End- and Field routines can be implemented in SQL Script

Intermediate results can be displayed in the DTP simulation mode

PERI7 conversion exit is supported

DTP with error handling is supported

ADSO with NLS-IQ archive is supported as a source


BW on SAP HANA Transformations – Debugging I

Initialize Debugging

To debug a SAP HANA Script it is necessary to execute the DTP in the Execution mode (not in the simulation mode)

Initialize Debugging

To debug a SAP HANA Script it is necessary to execute the DTP in the Execution mode (not in the simulation mode)

Debug Perspective

To debug the procedure it is necessary to switch to the Debug Perspective

Debug Perspective

To debug the procedure it is necessary to switch to the Debug Perspective


BW on SAP HANA Transformations – Debugging II

Call Stack Information

Current processing line number

Call Stack Information

Current processing line number

Source Code

Current processing source code

Add / Delete breakpoints

Source Code

Current processing source code

Add / Delete breakpoints

Variables

Available variables

Values for scalar variable

Number of lines for table type variables

Variables

Available variables

Values for scalar variable

Number of lines for table type variables

Data Preview

Preview for table type variables

Data Preview

Preview for table type variables

Check out these blogs for more details:

http://scn.sap.com/community/data-warehousing/bw/blog/2016/05/24/hana-based-bw-transformationhttp://scn.sap.com/community/data-warehousing/bw/blog/2016/06/17/hana-based-transformation-deep-divehttp://scn.sap.com/community/data-warehousing/bw/blog/2016/06/23/hana-based-bw-transformation--analyzing-and-debugging

http://scn.sap.com/community/data-warehousing/bw/blog/2016/05/24/hana-based-bw-transformation

http://scn.sap.com/community/data-warehousing/bw/blog/2016/06/17/hana-based-transformation-deep-dive

http://scn.sap.com/community/data-warehousing/bw/blog/2016/06/23/hana-based-bw-transformation--analyzing-and-debugging

http://scn.sap.com/community/data-warehousing/bw/blog/2016/06/23/hana-based-bw-transformation--analyzing-and-debugging



Conclusions how to approach design / development

Use standard push-down capabilities (only use SAP HANA Expert Script if necessary)• allows the execution in ABAP as well as SAP HANA easier for functional debugging; later SAP HANA as performance

mode

• profit from general improvement of the HAP framework

• transferring ABAP code to SQL Script in AMDPs requires careful review from performance perspective

• new skillset for programming / debugging / performance analysis required (new debugging features with SAP HANA SP10)

• expert routine was often used to avoid “overhead” of the generated program no driver for SAP HANA Expert Script

• please review Note 2057542 - Recommendation: Usage of SAP HANA-based Transformations

Avoid the usage of many Info Sources for one DTP• embedded calculation scenarios create complex plans which are complicated to analyze

• even optimized plans might contain more operations than necessary performance drain

Public


SAP HANA

SAP BW

Re

po

sit

ory

Sources

BI Clients

Data Warehousing

SAP HANA Views

gen.SAP

HANA Views

SAP HANA Views

Data Acquisition

SAP HANA

SAP BW

Re

po

sit

ory

Sources

Data Warehousing

gen. SAP

HANA Views

Data Acquisition

Analytic EngineBW Query BW Query

gen.SAP

HANA Views

Wo

rks

pa

ce

s

BI Clients



Two options available:

• Pull mechanism via SAP HANA modeler (introduced with BW 7.30)

• Push mechanism triggered from BW (available since BW 7.40)

how can you transform your architecture from pull to push (*)?

(*)TechEd 2015 (DMM301) focused on maintenance aspects as well as feature coverage (Calculation View via Modeler vs. Composite Provider / External Query View)Push mechanism simplifies governance and allows simple adoption of new BW features. New WEB Modeler and BW Edition for SAP HANA will not support Pull mechanism.



Pull: is manually triggered from SAP HANA modeler

• An Analytic View for the selected Info Providers will be created in a specified package

• Optionally classic (XML-based) Analytical Privileges can be created (view always applies privileges)



Push: Info Provider property; view generated at Info Provider activation

• Analytic View will be created in a globally specified package at Info Provider activation• Generated View secured by SQL Analytic Privileges (optionally SQL-APs can be generated

based on BW analysis authorizations)



Push: Info Provider property; view generated at Info Provider activation

Optional calculation view deployment configurable via transaction RS2HANA_VIEW

Note: generated calculation view directly dependent on underlying base tables for classic DSO and Info Cube



How do pull architectures typically look like?

• Pull mechanism is able to create content objects from physical Info Providers (Info Cube, classic DSO, Info Object, Query as Info Provider)

• SAP HANA modeler is used to create additional logic based on calculation views (views potentially stacked, unions / joins, calculated / restricted measures)



How to perform the transition? There are different options:

• Direct exchange of Part Providers

• Rebuild using Composite Providers + consumption in Modeler CVs

• Rebuild using Composite Providers + consumption in BW Query (external query views)



Direct exchange of Part Providers

• The modeler provides functionality to substitute the data sources of the calculation view

• Text columns need to be remapped manually since the naming convention differs between pull and push views



Direct exchange of Part Providers

• Minimal manual effort (reuses old models but not fully governed by BW higher maintenance efforts)

• Not feasible in all cases (e.g. no push for Part Providers of SPO possible expose via Composite Provider)



Rebuild using Composite Providers + consumption in Modeler CVs

• Create a new Composite Provider and consume the relevant Info Providers as required

• In the SPO case create an empty Composite Provider and add the SPO as a data source




• Assignments can be created analog to other Part Provider




• Composite Provider is exposed as a Calculation View in the global package




Difference to direct substitution of Part Providers:

• Datasources of CP scenario cannot be deployed as calculation views (scenarios) for classic DSOs and Info Cubes (performance consideration, e.g. join push-down)

• For other BW providers the external view uses the 0BW:BIA view (internal logical representation) as a datasource (therefore no difference as both views are identical)




• Replace Calculation View Node with a Data Source (here: Composite Provider external view)




• Remap text columns due to different naming convention



Rebuild using Composite Providers + consumption in BW Query (external query views)

• If Composite Provider and Query designer features can represent original model a full remodeling governed by BW is possible




• External View for BW Query is exposed in a package named after the underlying Info Provider




• The external query view consists of a single column view

• The associated calculation scenario uses the Composite Provider as a datasource and computes the layered query on top




• In addition to SQL access via the external view, the access via BW BICS is possible with this modeling approach as well



Pull PushPart Providers of SPOs can be pulled SPO can be exposed as external view via Composite Provider

Modifications possible (calculated attributes, calculated/restricted measures) are retained at reimport

Only overwrite possible

Package selection available at import Fixed package for all imports

Optional inclusion of display attributes Display attributes not available (switch to NAV necessary)

No NLS connection NLS connection available (regarding pruning refer to Note 2190204)

Advanced Data Store Object (ADSO) not available ADSO push possible (NAV attributes via Composite Provider – Note 2215947)

Not available on Data Provider level Non-cumulative KFs via External Query View

Classic (XML-based) Analytic Privileges SQL Analytic Privileges (automatic generation of BW authorizations as APs)

Request Handling (QUALOK) independent Request Handling implemented via generated BW authorizations

Always based on Analytic Views Deployment as Calculation View possible

Differences between Pull and Push content generation

Comparison of the Two Methods for Creating Views in SAP HANA for BW Objectshttp://help.sap.com/saphelp_nw75/helpdata/en/17/8745cfaeed428eb1fa6d86ad2e40e6/content.htm



How to migrate authorizations?

• Pull Views are secured by Classic APs

• Push Views employ the newer SQL APs

• Both checks cannot be combined on a single view

• How to migrate an existing freestyle authorization concept (without additional BW analysis authorizations)?



How do the generated analysis authorizations for PUSH views look like?

Created external view secured by SQL Analytic Privileges

Analytical Privilege created for the Composite Provider

Automatically assigned to DBMS User

BW table RS2HANA_AUTH_STR contains filter values according to authorizations for current SESSION_USER

Mapping between BW User and SAP HANA User is maintained in transaction SU01 – DBMS Tab




• Per default BW analysis authorizations will be generated at Info Provider activation / transport

• This can be prevented in transaction RS2HANA_VIEWif necessary only view with SQL AP check is created

• If needed manual replication can be performed via:Report RS2HANA_AUTH_RUN (transaction RS2HANA_GEN)Process chain (useful since authorizationchanges in BW are not automaticallypropagated to SAP HANA side)




• SQL APs need to be created manually analog to classic APs (no conversion routine available)

• SQL APs are assigned to SQL users and will be checked at runtime



Conclusions / Short Summary

• Transformation to push on physical Info Provider level or Composite Provider / External Query View level possible

• Selection of architecture depends on• modeling feature & maintenance requirements (DMM301, TechEd 2015)• backend architecture (SPOs, DIS/NAV attributes, ADSO usage, privileges)• project effort

Public


SAP HANA

SAP BW

Re

po

sit

ory

BI Clients

Data Warehousing

Data Acquisition

Analytic EngineBW Query

Composite Provider

SAP HANA Views

Sources

SAP HANA

SAP BW

Re

po

sit

ory

Sources

BI Clients

Analytic EngineBW Query

Composite Provider

SAP HANA Views



Why to integrate SAP HANA Views?

Data is located partly or completely outside of BW

Data might have been processed / produced by any SAP HANA native application

Integration into BW is desired to:

• combine data for subsequent processing (e.g. planning)

• use features dedicated to the analytic engine

• create a persistence in BW



How to integrate SAP HANA Views (or external data)?

Two new providers are available in BW 7.40

Composite Provider

• combines data from BW Info Providers (Info Object, DSO, SPO, Info Cube, SAP HANA Views, Open ODS Views) for reporting

Open ODS View

• integrates / conforms data from external objects like database tables, database or SAP HANA views, virtual tables or BW Datasources with direct access

• can generate ADSO (Advanced Datastore Object) as persistence in BW



Modeling options 1) Integration of database table, database view or virtual table only possible viaOpen ODS View

2) Integration of SAP HANA View into Composite Provider via Open ODS View

3) Direct Integration of SAP HANA View into Composite Provider



Performance Implications

Conforming models to BW (e.g. ALPHA conversion, no integer data type for characteristics)

• basic conversions available in Open ODS View (calculation scenario since BW 7.40 SP8)

• SQL View not recommended as it can disrupt query optimization especially before SAP HANA SP10

• more complex conversions can be explicitly modeled in a calculation view (cast operations with calculated columns) might prevent filter push-down / causes larger intermediate results

• conversion during staging / SAP HANA optimized transformations



Conclusions / Short Summary

Goal: external data should be integrated into BW

Two Info Providers available: Open ODS View and Composite Provider

Consider functional aspects (feature-driven) which Provider to use

Performance aspects influence the modeling approach

• additional encapsulation of information view into Open ODS View might be beneficial

• consider effects of conforming data types (Open ODS View Conversions to staging solution via SAP HANA enabled transformations)


Design and Performance considerations SAP HANA Composite Providers

SAP HANA Composite Provider

Null Value handling

Referential Integrity

Compounding problem

Key-based processing

Navigation Attributes

Fields w/o InfoObject Association

Additional Joins during Query Execution

Local SID limitations

Value help for Fields

No SAP HANA exception aggregation pushdown ( <= BW 7.40)

SAP HANA pushdown operations may not be possible

No SAP HANA pushdown of exception aggregation

No pushdown of any hierarchy filter

BW handles NULL values as initial values. Therfore filter on inital values implicitly includes a filter on NULL

No OLAP Cache

No parallel processing

Avoids Null Value handling for SAP HANA models (except Left Outer Join Scenarios). see also Null value handling

No exception aggregation in case several source fields are mapped to one target field

Nearline StorageNo NLS in case HCPR runtime

is required (e.g. Joins, Navigational Attribute, etc.)


Design and Performance Guidelines I

The design considerations are referencing to SAP Note 2271658 The following aspects need to be considered to gain best possible query performance against composite providers

Navigation Attributes Switch on required Navi-Attributes on PartProviders to avoid the generation of an additional CalcScenario

which gets joined to the HCPR during Query Execution

Null Value Handling ABAP App Server cannot distinguish between initial value and NULL. NULL values can occur in HCPRs with Left Outer Joins or in HCPRs which consume SAP HANA models.

No Pushdown of exception aggregation No Pushdown of hierarchy filters

To avoid special NULL value handling for SAP HANA models is to guarantee that master data exisits for any transactional record and referential integrity is flagged.


Design and Performance Guidelines II

Fields without InfoObject Association Local SIDs are generated during query Execution for fields without InfoObject Association or Associations to

OpenODS Views No OLAP cache usage possible No parallel processing in the datamanager

Usually value helps reads masterdata of the associated InfoObject Without association value helps tries to read data from HCPR

- In Case HCPR contains CalculationViews with mandatory InputParameters it is only possible to read the data from HCPR if for all mandatory InputParameters a default value exists. Otherwise no data is shown

Compounding Problem See note 1045683 for compounding-issue description. Add missing InfoObjects to PartProvider to overcome the Compounding-issue


Design and Performance Guidelines III

Referential Integrity The Flag ‚User confirmed Referential Integrity‘ must only be set if master data values exist in BW for any

transactional record returned by the HCPR. Otherwise the result set of a query might be filtered by the existing values in the BW master data tables

Referential Integrity in a HCPR with SAP HANA models may be beneficial to avoid the special handling of NULL values (see also Null value handling)

No exception aggregation push down in case several source fields are mapped to one target field.

NearLine Storage NLS requires a separate data access and is not embedded directly in the CompositeProvider runtime,

therefore NLS can only be used if the HCPR runtime is not required. Joins, navigation attributes which cannot be read from relevant PartProvider, etc. require HCPR runtime

Key-based processing SAP HANA exception aggregation pushdown with (BW <= 7.40) only possible if master data SIDs are

available in the InfoProvider or by joining the master data SID table


SAP TechEd Online

Continue your SAP TechEd education after the event!

Access replays of Keynotes Demo Jam SAP TechEd live interviews Select lecture sessions Hands-on sessions …

http://sapteched.com/online

http://sapteched.com/online


Further information

Related SAP TechEd sessions:DMM104 - SAP HANA Data Warehousing: Overview, Components, and Future Strategy Lecture (1hr)DMM162 - SAP HANA Data Warehousing: Build and Run a SQL Data Warehouse on SAP HANA Workshop (2hr) DMM203 - New Approaches for Data Modeling with SAP HANA Lecture (1hr)DMM260 - Introduction to Data Modeling in SAP HANA Workshop (2hr) DMM265 - SAP HANA Data Warehousing: Simplified Modeling with SAP BW 7.5 SP4 Workshop (2hr) DMM268 - SAP HANA Data Warehousing: Mixed Scenario for SAP BW and SQL DW on SAP HANA Workshop (2hr) DMM269 - End-to-End Model Performance Analysis in Native SAP HANA Platform Scenarios Workshop (2hr) DMM302 - SAP HANA Data Warehousing: Models for SAP BW and SQL DW on SAP HANA Lecture (2hr)DMM360 - Advanced Data Modeling in SAP HANA Workshop (2hr)

SAP Public Webscn.sap.com www.sap.com

SAP Education and Certification Opportunitieswww.sap.com/education

Watch SAP TechEd Onlinewww.sapteched.com/online

http://scn.sap.com/welcome

http://www.sap.com/

http://www.sap.com/education

http://www.sapteched.com/online


Thanks for attending this session.

Please complete your session evaluation for DMM300

Contact information:

F name MI. L nameTitleemail address

Feedback


© 2016 SAP SE or an SAP affiliate company. All rights reserved.

No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP SE or an SAP affiliate company.

SAP and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP SE (or an SAP affiliate company) in Germany and other countries. Please see http://www.sap.com/corporate-en/about/legal/copyright/index.html for additional trademark information and notices.

Some software products marketed by SAP SE and its distributors contain proprietary software components of other software vendors.

National product specifications may vary.

These materials are provided by SAP SE or an SAP affiliate company for informational purposes only, without representation or warranty of any kind, and SAP SE or its affiliated companies shall not be liable for errors or omissions with respect to the materials. The only warranties for SAP SE or SAP affiliate company products and services are those that are set forth in the express warranty statements accompanying such products and services, if any. Nothing herein should be construed as constituting an additional warranty.

In particular, SAP SE or its affiliated companies have no obligation to pursue any course of business outlined in this document or any related presentation, or to develop or release any functionality mentioned therein. This document, or any related presentation, and SAP SE’s or its affiliated companies’ strategy and possible future developments, products, and/or platform directions and functionality are all subject to change and may be changed by SAP SE or its affiliated companies at any time for any reason without notice. The information in this document is not a commitment, promise, or legal obligation to deliver any material, code, or functionality. All forward-looking statements are subject to various risks and uncertainties that could cause actual results to differ materially from expectations. Readers are cautioned not to place undue reliance on these forward-looking statements, which speak only as of their dates, and they should not be relied upon in making purchasing decisions.

http://www.sap.com/corporate-en/about/legal/copyright/index.html

dmm300 – mixed scenarios for sap hana data warehousing and bw: overview and experiences

Data & Analytics