dmm203 – new approaches for data modelingwith sap hana

Public

DMM203 – New Approaches for Data Modeling with SAP HANA

© 2016 SAP SE or an SAP affiliate company. All rights reserved. 2Public

Speakers

Bangalore, October 5 - 7

Christoph Morgen

Las Vegas, Sept 19 - 23

Christoph Morgen

Barcelona, Nov 8 - 10

Christoph Morgen


Disclaimer

The information in this presentation is confidential and proprietary to SAP and may not be disclosed without the permission of SAP. Except for your obligation to protect confidential information, this presentation is not subject to your license agreement or any other service or subscription agreement with SAP. SAP has no obligation to pursue any course of business outlined in this presentation or any related document, or to develop or release any functionality mentioned therein.

This presentation, or any related document and SAP's strategy and possible future developments, products and or platforms directions and functionality are all subject to change and may be changed by SAP at any time for any reason without notice. The information in this presentation is not a commitment, promise or legal obligation to deliver any material, code or functionality. This presentation is provided without a warranty of any kind, either express or implied, including but not limited to, the implied warranties of merchantability, fitness for a particular purpose, or non-infringement. This presentation is for informational purposes and may not be incorporated into a contract. SAP assumes no responsibility for errors or omissions in this presentation, except if such damages were caused by SAP’s intentional or gross negligence.

All forward-looking statements are subject to various risks and uncertainties that could cause actual results to differ materially from expectations. Readers are cautioned not to place undue reliance on these forward-looking statements, which speak only as of their dates, and they should not be relied upon in making purchasing decisions.


Agenda

• SAP HANA View Modeling

– Modeling Graphical Calculation Views today

– Converting your Views to Calculation Views

• New SAP HANA Data Modeling Approaches

– Developing with the SAP Web IDE for SAP HANA

– Graphical development of CDS entities

– Web-based Calculation View development enhancements

– Using CDS Views with Calculation Views

– SAP PowerDesigner Modeling for HANA

Public

SAP HANA View Modeling

Today’s approach of modeling Graphical Calculation Views


SAP HANA View Modeling – Overview

Virtual Data Modeling as a key SAP HANA concept

SAP HANA PLATFORMSAP HANA PLATFORM

Database LayerDatabase LayerPhysical TablesPhysical Tables

HANA ViewsHANA Views

Virtual Data Flow Models

• No persistent aggregations | single atomic copy of data

• In-memory engines | performance

• Multidimensional Reporting Models

• Enterprise Applications Virtual Data Models

Operational Reporting | Applications | Analytics



Evolution of SAP HANA View Modeling Approaches

Attribute View

Analytic View

Calculation Views(Graphical and Script-based)

Graphical Calculation Views*

*and SQLScript-based Table Functions as data sourceAny of the approaches is still fully supported as of today with SAP HANA SPS12 !



Modeling Graphical Calculation Views

The Calculation View Modeling Toolbox Calculation Views of type Dimension

– Composing Dimension-Attributes and -Hierarchies

– Has no measures, does not aggregate

– Creates BI View-metadata / exposed to BI-Tools

Calculation View of type CUBE– Has measures, always aggregates

– Creates BI View-Metadata / exposed to BI-Tools

Calculation Views of type CUBE with StarJoin– Star-joins fact data with dimension-type calculation views

– Creates BI View-Metadata / exposed to BI-Tools

Calculation View of type “<default>” / “<empty type>”– Free-form, custom calculation view

– Does not create BI View-Metadata / not exposed to BI-Tools

Dimension

Cube Star-Join

Cube




The Calculation View Modeling Toolbox Compose a dataflow scenario using node-operations

– Projection, Join, Union, Aggregation, Star-Join, Rank and Graph*

Select from a range of data sources– SAP HANA Tables (columnar, …),

CDS entities (design-time data sources)

– SQL Views

– SQLScript Table Functions

– Remote SAP HANA database container objects

– Remote smart data access / virtual data sources

Stacking of Calculation Views to larger data models– Link and mapping of parameters between Calculation Views

– Sharing, extraction and propagation of semantic information

*New only within the SAP Web IDE for SAP HANA development envrionment.


SAP HANA View Modeling – Key Concepts

Modeling multidimensional Scenarios in Calculation Views

Multidimensional scenarios in Calculation Views Calculation Views of type Dimension

– Composing Dimension-Attributes and -Hierarchies

Calculation Views of type CUBE with StarJoin– Star-joins fact data with dimension-type calculation views

like Analytic Views (with same performance guidance's)– Implicit Calculation View optimization will evaluate

to delegate processing to same execution behavior as Analytic Views

Usage Scenario Implement new multidimensional scenarios / new projects Seek to refactor Attribute-/Analytic View scenarios

Calculation Views

Note: As of SAP HANA SPS12 Calculation View Star-Join scenarios cover Analytic Views capabilities incl. Temporal Joins



Modeling multidimensional Scenarios in Calculation Views

Hierarchy capabilities in Calculation Views Parameter use within Hierarchies

– Use within calculated columns of hierarchies– Setting Hierarchy root-nodes

Time dependent hierarchies

Hierarchy-based SQL processing of Views

– Hierarchy-based filtering and aggregations

Hierarchy-based Variables

– Hierarchy-based value help and filtering

Hierarchy-driven SQL-based Privileges

NotAssigned- and NULL value handling in Hierarchies

Full composite key support for Parent-Child Hierarchies



Modeling with Virtual and Extended Tables

Consuming non-In Memory data in Calculation views Dynamic Tiering

– Provides a disk-based columnar SAP HANA database extension usingSAP HANA external tables (extended tables) *

Smart Data Access– Provides a virtual access layer to outside SAP HANA data (e.g. other

databases, Hadoop systems, etc.) using so-called virtual tables

Leveraging “warm data” from SAP HANA Dynamic Tiering external tables

Leveraging “cold data”

via SAP HANA Smart Data Access

virtual tables

Model and approach these scenarios carefully and monitoring query performance– Remote data imposes a natural break in the execution – Ensure that filters and aggregation is pushed to remote sources– Use and observer join settings and optimizations– Implement input source pruning (explicit or implicit) techniques

*Supported external systems and restrictions are documented in SAP Note 1868209

https://css.wdf.sap.corp/sap/support/notes/1868209

© 2016 SAP SE or an SAP affiliate company. All rights reserved. 13Public© SAP AG 2011

Enabling SQLScript with Calculation views• SQLScript Table Functions allow to embed custom scripting

– Side-effect free read-only functions using standard SQLScript– Consume data from tables, models, functions, predictive algorithms, etc.– Script based-calculation models can be re-factored into table functions


Using SQLScript with Graphical Calculation Views

Graphical Calculation View referencing Table Function



Using SQLScript with Graphical Calculation Views

© SAP AG 2011

SQLScript approaches• Standard SQL versus Calculation Engine (CE Functions) Plan Operators

Recommendation! When creating functions or procedures with SAP HANA SPS09+ use standard SQL syntax instead of CE Functions. The SQL optimizer is able to leverage alternative execution engines and are not restricted only to the Calculation Engine

Recommended approach



Expression Language in Calculation Views – SQL Expressions

General SQL support within Expression Editors Explicit expression language selection

– Explicit distinction between Column Engine expressionand SQL Expression was introduced for Calculation Columns since SAP HANA SPS10

– Filter expressions– Restricted measure expressions– Default value expressions for Variables/Parameters

Expression language conversion behavior– Harmonized execution modes between Column Engine-

and SQL expression language– No result differences between use of a SQL vs Column

Store expression due to respective semantic functions conversion

Example – Add sematic using a NULL-value

Within a column store expression, you want to use the column store semantic addition behavior 1 + NULL = 1

This may implicitly get converted into a SQL expression ADD_NAZ(1,NULL) = 1

Within a column store expression, you want to use the SQL semantic addition behavior SQLADD(1,NULL) = NULL

This may implicitly get converted into a SQL Function 1 + NULL = NULL

SAP NOTE: 1857202 - SQL Execution of calculation views



Analytic Privileges Editor – SQL-based Analytic Privileges

Create SQL-based Analytic Privileges Start with general Attribute based Analytic Privilege, then switch to SQL-based

Use SQL Hierarchies within SQL Analytical Privileges

SalesHierarchyNode = MAJESTIX SalesHierarchyNode = MAJESTIX


Additional SQL Optimization


Optimized Calculation View Execution

Since SAP HANA Rev 62 the SQL optimization can explicitly be determined settingthe “Execute In:” property to “SQL Engine”

Since SAP HANA SPS9 the optimized execution behavior implicitly evaluates to leverage additional SQL optimization

SAP HANA SPS10 optimization enhancements include implicit optimized Star Join processing

Optimized execution for SQL-queries against Calculation Views The initial Calculation Engine optimization generates a single SQL statement across a stacked model,

which is then passed to the SQL optimizer The SQL optimizer adds additional optimizations and delegates operations to the best database execution

operator, e.g. delegate star join-nodes to the OLAP processing engine where possible

Calculation Engine Optimization

Calculation Views

SQL Optimization includes

• Stacked Views /SQL unfolding

• Cost-based optimization

• Delegate processing to best operator

* Implicit SQL optimization details are documented in SAP Note 2223597 and 2291812.

https://css.wdf.sap.corp/sap/support/notes/2223597

https://launchpad.support.sap.com/#/notes/2291812


Visualize PlanCalculation View Debugger


Performance Analysis & Query Insight Tools

Timeline – root cause analysis

Operator list

Explain Plan




Advantages of the Calculation View Modeling approach Benefit from better optimization

– Queries against stacked models of only Calculation-Views,provide better overall optimization options

– Better pruning / filter–push down– Less materialization / no View-types changes– Table Functions included in overall optimization

e.g. Filter push-down

Modeling Flexibility– Componentize larger data models,

create re-usable partial models and stack– Map parameters and variable between views

Leverage Performance Analysis tools– Calculation View Debugger

Stacked Calculation View Data Model


SAP HANA View Modeling – New Approaches

Converting all SAP HANA Views to Calculation Views

Going forward with Graphical Calculation Views Graphically modeled Calculation Views target to unify all SAP HANA View capabilities,

with respect to functionality as well as performance Convert within the classic SAP HANA Repository

Attribute Views

Analytic Views

Script-based CVs

CVs

Modeled Analytic Privileges

Calculation Views

Table Functions

SQL-based Analytic Privileges

Convert using the HANA Studio Migration Tool

Classic HANA Repository objects*

*this conversion is a preparation step before migrating to SAP Web IDE for SAP HANA.




Going forward with Graphical Calculation Views Converting your SAP HANA Views

– Dimension-Type graphical Calculation Views replacing Attribute-Views– Cube-/StarJoin-Type graphical Calculation Views replacing Analytic-Views– Table Functions as Data Sources in graphical Calculation Views

replacing Scripted-Calculation Views– SQL-based Analytic Privileges replacing classic modeled Analytic Privileges

With SPS11 a migration utility is initially provided to support the conversion process




How to find SAP HANA documentation on this topic?• See SAP Note https://launchpad.support.sap.com/#/notes/2325817



Public

New Approaches

SAP HANA Data Modeling – New Approaches



Graphical Development Tools – Innovation Path

SAP HANA Web-based Development Workbench Database & Application Development Initial releases of graphical development tools

SAP HANA Studio – Modeler Eclipse-perspective Primarily focused on HANA View modeling artifacts

SAP HANA Studio – Development Eclipse-perspective Support for all application and database design-time

artifacts and providing additional repository functionality– E.g. Script-based CDS development, flowgraphs, ….

SAP Web IDE for SAP HANA

New Development tooling platform for SAP HANA, initially delivered in SPS11*major release with SAP HANA SPS12

All development tools– (re-) implemented as

XS Advanced applications leveraging new UI frameworks of the SAP Web IDE for SAP HANA and

– commonly building on the new HANA deployment infrastructure.

Web-based graphical editors– Calculation Views– CDS entities– Data flows– Lineage and impact analysis

*SAP Note 2304873 SAP Web IDE for SAP HANA SPS12 - Central Release NoteDEV204 – What's New in Native Application Development with SAP HANA DEV161 – Intro to SAP HANA Extended Application Services, Advanced Model Development



The new Web-based HANA Development Environment

SAP Web IDE for SAP HANAthe new and future development environment for SAP HANA• Using MTA Projects (Multi-Target Application)• Using HDB Modules

An isolated HANA Deployment InfrastructureContainer (~ isolated schema as your HANA development workspace)

Any access is executed via (and restricted to)a HDI-container-specific technical user, NOT by your XSA webIDE development user

There is no default access to other schemas on the same system

Namespaces can be defined, to support name-structuring all development artefacts

*_HDI_CONTAINER SCHEMA YXZ„external data“

_SYS_BI

DEV160 – Intro to SAP HANA Database Development with Deployment Infrastructure



SAP HANA Graphical CDS Editor – Overview

Graphical development of CDS objects• Graphical CDS Editor• Supported capabilities incl.

Contexts Entities, incl. HANA native types Structured Types Indexes, incl. Search/Full Text Indexes Table Groupings, Partitioning Associations



SAP HANA Graphical CDS Editor – Overview

Graphical development of CDS objects• Graphical CDS Editor• Supported capabilities incl.

Contexts Entities, incl. HANA native types Structured Types Indexes, incl. Search/Full Text Indexes Table Groupings, Partitioning Associations References to CDS-Views



SAP HANA CDS Development – Graphical & Script-based Editor

CDS development editor choices• Graphical and Script-based CDS editors• Graphical editor

opens any hdbcds-file Persist any non-

graphically supported capability

• Script-based editor opens any

graphically developedhdbcds-file



The new Graphical Calculation View Editor – Overview

New Web-based Editor for Calculation Views• Supports all major capabilities for

graphical Calculation View Development

• Supported capabilities incl. Joins, Unions, Projections, Aggregations,

StarJoins (incl. temporal join), Rank Variables, Input Parameters, Calculated

Columns/Filters (incl. SQL expressions), Unit &Currency Conversion

Catalog metadata services, Initial convenience / supportability functions

e.g. local refactoring, basic data preview, performance analysis

• New capabilities incl. Cross-object impact and lineage analysis Graph analysis modeling



Web-based Calculation View Editor – Data Sources

Calculation View Data Sources

• Data Sources from within your HDI container Design-Time objects local to your

development container

Tables, Table Functions, SQL Views, Calculation Views, etc. are supported inputs

• Data Sources from outside your HDI container Synonym definitions are required to

reference to data sources outside of your current development container



Web-based Calculation View Editor – Star Joins / Temporal Joins

Modeling Star Joins Create graphical calculation views

– Join multiple dimensions with thefact data input set.

– Now, inner-join use supportstemporal Join-conditions(to match Analytical View capabilities)

Star Join - Join Definition incl. details (Referential, LeftOuter, …)



Web-based Calculation View Editor – Deployment and Access

Target Deployment concept

• Like the development container, the deployment target is as well an isolated HDI container

• New -> Deployment to any Schema– Calculation Views are no longer deployed to _SYS_BIC,

but can be deployed to “any” HDI-container (“schema”)

• New -> Security and Data Access requirements– XSA default approach to container access is via a technical

user + business user information passed alongo Data analytic privileges shall be managed on HANA

o User-specific object privileges shall be managed by the App

– Non-XSA application / BI Clientso Grant privilege/container role to access the container's schema

o Grant privilege to read BIMC tables.

*_HDI_CONTAINER SCHEMA YXZ„external data“

_SYS_BI

o Cross schema data access provided by synonynm and a XSA data access service to be defined

SEC161 - Learn from the Experts: Secure and Extend Advanced SAP HANA Applications (Hands-On Workshop)



Graph Data Modeling and Processing

SAP HANA GraphGraph data processing capabilities to allow businesses to map connections between entities in a network and discover relationships on the fly.

• Property Graph Model with flexible schema with ability to traverse relationships without the need for predefined modeling

• Built-in high-performant graph algorithms (neighborhood search, shortest path, strongly connected components, pattern matching) to find connections without manually creating complex JOIN statements

• Graph viewer tool - allows for quick visualization and dynamic interaction (i.e. change algorithm parameters) with graph data in real-time

• Graph Analysis Modeling – Integrated with SAP Web IDE for SAP HANA Calculation View Editor for creating graph analysis models visually instead of via SQL or SQLScript S A P H A N A

SAP HANA Graph

Property Graph Model

Graph EngineNative

Algorithms

Graph Viewer Graph Modeler



Modeling a SAP HANA Graph Analysis

Create Edge and Vertex Tables

With as many attributes as you want

Workspace

Relationships Members

Create Graph Workspace



Modeling a SAP HANA Graph Analysis

Create Graph Analysis Model

Query the Graph Analysis Model

Shortest Path



SAP HANA Lineage- and Impact Analysis

New Web-based Data Modeling Editors

• New capabilities incl. Cross HANA-object Lineage and Impact analysis



Converting all SAP HANA Views to HDI Development Objects

Step 1 - Unified object conversion

Convert using the HANA Studio

Migration Tool

Going forward with Graphical Calculation Views in SAP Web IDE for SAP HANA If you seek to move and migrate your SAP HANA development towards SAP Web IDE for SAP HANA,

classic HANA repository development objects, need to be converted to HDI-based development objects

–

Attribute Views

Analytic Views

Script-based CVs

CVs

Modeled Analytic Privileges

Calculation Views

Table Functions

SQL-based Analytic Privileges

HDI-based design-time artefacts

• Calculation Views

• Table Functions

• SQL-based Analytic Privileges

Step 2 - HDI-object conversion

HDI-migration/conversion tools are planned to be delivered with a future shipment after SPS12 to help move your applications from XS Classic

and the HANA Repository to XS Advanced and HDI.

DEV703 – Migrating Apps Between Models of SAP HANA Extended Application Services.


New Data Modeling Approaches

SAP HANA Core Data Services

SAP HANA Core Data Services (CDS) Design-time language for defining models (tables, views)

as part of native HANA applications in XSA/HDI (or XSC/Repository) Targets Application data models

HANA CDS capabilities include– Data Control Language, User defined types, Associations,

Annotations, Extensibility, … – Source based definition

-> deployment managed by HDI (or HANA Repository)– Runtime catalog objects

-> SQL defined tables and views– Lifecycle management– Data model is located in the database and

not part of an application stack or framework on top

HANA CDS ≠ ABAP CDS

HANA Schema with table, views

HANA Schema with additional CDS metadata

SQL Engine

CDS CDS Catalog

Schema XYZ



Modeling Views – Using CDS or Calculation Views?

SAP HANA Core Data Services Views CDS views technically are SQL views “Limited” to SQL exposed HANA functionality No graphical editor for CDS views currently “Simple” views should behave similarly w.r.t. performance Do not support analytic annotations

Using one or the other ? Analytics / BI-use case

– Calculation View provide rich BI metadata (cubes, hierarchies, ….)for SQL / MDX, SAP and 3rd –party BI Tools

– ABAP CDS views support analytic annotations (analytic query), consumption requires NetWeaver “analytic query layer” e.g. consumable by Design Studio + AO / 3rd party NetWeaver-based BI tools (MDX)




Using one or the other ?

Performance aspects

–Calculation View options provides additional pruning and optimization during their runtime instantiation process (SAP Note 2223597), thus likely to provide better performanceoAll columns not requested by the query are removed from the instantiated calculation model

o In addition complete join/union data sources might be pruned during instantiation

–Avoid mixing Calculation Views and CDS ViewsoTo ensure best optimization, do not consume Calculation Views from CDS Views

oSimple CDS Views as data sources to Calculation View can be used




Using one or the other ?

How important is the tooling / development approach–Graphical or scripting?

What‘s your Application scenario?–Building a distinct application or

mass development of analytic-structuresfor BI Tools?

Functional aspects?

How important is performance?CDS

Basic persistencyTables & SQL views

CDS

Basic persistencyTables & SQL views

Calculation Views on top ofCDS tables/SQL views

Calculation Views

Analytical scenarios

Hierarchies, GraphEngine

Calculation Views

Analytical scenarios

Hierarchies, GraphEngine



SAP PowerDesigner – CDS Modeling Support

CDS Modeling Support in SAP PowerDesigner* Physical Data Model generation as HANA CDS

– Define CDS entity and Types– Define CDS Views– Generate/activate CDS artifacts in HANA o CDS Repository Artifacts generated and activated in the repositoryo CDS HDI: Artifacts generated as .hdbcds files

DMM111 – Modeling the End-to-End Data Warehouse with SAP PowerDesignerDMM101 – Architecting Your Enterprise with SAP PowerDesigner

*SAP PowerDesigner 16.6 SP01



SAP PowerDesigner – Calculation View Modeling Support

Calculation View Modeling support in SAP PowerDesigner

View specification describes Output Projection and Sources– Calculation view defined in HANA Catalog PDM

– Generate Calculation-view skeleton in HANA (Projection and sources)

– Simple Calculation views based on one or two data-sources can be generated and activated

– Partially reverse-engineer Calculation-views (data-sources and input columns for impact-analysis purpose)

Using Calculation-views with CDS entities (HDI)– Calculation-views and CDS artifacts defined in different models

– CDS artifacts must be first generated (and activated), then reversed as tables before being used as Calculation-view data-sources.

*SAP PowerDesigner 16.6 SP01


SAP HANA Data Modeling Key Concepts and New Approaches

Your SAP HANA View Modeling path forward

You seek to run for longer on SAP HANA SPS12– Model Calculation Views in SAP HANA Studio

– You don’t have migrate Attribute- and Analytic Views in order to run on SPS12

– It’s a chance to optimize

– New HANA View development should be focused on Calculation Views

– If you build XSA Applications on SPS12, explore SAP Web IDE for SAP HANA Data Modeling tools

You are ready to jump or require to move fast to the next SAP HANA release– Prepare to convert all SAP HANA Views to Graphical Calculation Views

– Start and explore SAP Web IDE for SAP HANA Data Modeling tools

– Familiarize with the new overall development approach

– Validate current feature set, determine if you can start on SPS12


SAP TechEd Online

Continue your SAP TechEd education after the event!

Access replays of Keynotes Demo Jam SAP TechEd live interviews Select lecture sessions Hands-on sessions …

http://sapteched.com/online

Recommended TechEd 2015 Session DMM208: New and Best Practices for Data Modeling with SAP HANA - http://events.sap.com/teched/en/session/26543


Further information

Related SAP TechEd sessions:DMM260 – Introduction to Data Modeling in SAP HANADMM360 – Advanced Data Modeling in SAP HANADMM269 – End-to-End Model Performance Analysis in Native SAP HANA Platform ScenariosDMM302 – SAP HANA Data Warehousing: Models for SAP BW and SQL DW on SAP HANADMM162 – SAP HANA Data Warehousing: Build and Run a SQL Data Warehouse on SAP HANADMM300 – Mixed Scenarios for SAP HANA Data Warehousing: Overview and Experiences DMM111 – Modeling the End-to-End Data Warehouse with SAP PowerDesignerDEV204 – What's New in Native Application Development with SAP HANA DEV160 – Intro to SAP HANA Database Development with Deployment Infrastructure DEV161 – Intro to SAP HANA Extended Application Services, Advanced Model DevelopmentDEV703 – Migrating Apps Between Models of SAP HANA Extended Application Services.

SAP Public Webscn.sap.com http://scn.sap.com/community/developer-center/hana http://scn.sap.com/community/hana-in-memorywww.sap.com hana.sap.com https://blogs.saphana.com/2016/05/10/what-is-new-in-sap-hana-sps12/

SAP Education and Certification Opportunitieswww.sap.com/education https://www.youtube.com/user/saphanaacademy https://open.sap.com/courses

Watch SAP TechEd Onlinewww.sapteched.com/online

http://sapteched.com/online


Thanks for attending this session.

Please complete your session evaluation for DMM203

Contact information:

Christoph Morgen SAP HANA Product ManagementSAP SE | Dietmar-Hopp-Allee 16 | 69190 Walldorf | Germany

[email protected]

Feedback

http://scn.sap.com/welcome

http://scn.sap.com/community/developer-center/hana

http://scn.sap.com/community/hana-in-memory

http://www.sap.com/

http://www.saphana.com/

https://blogs.saphana.com/2016/05/10/what-is-new-in-sap-hana-sps12/

http://www.sap.com/education

https://www.youtube.com/user/saphanaacademy

https://open.sap.com/courses

http://www.sapteched.com/online

dmm203 – new approaches for data modelingwith sap hana

Data & Analytics