bw edw workshop
Post on 30-Sep-2014
519 Views
Preview:
TRANSCRIPT
PDEBW1PDEBW1
The Layered Scalable Architecture (LSA) for BW on CorporateThe Layered Scalable Architecture (LSA) for BW on CorporateScale (EDW)Scale (EDW)
Juergen Haupt, ArchitectSAP Intelligence Platform and NetWeaver RIG EMEA
Daniel Rutschmann, SpecialistSAP Palo Alto - Service
BW Layered, Scalable Architecture (LSA) for EDW
BI Maturity & Data Logistic11
Model-driven SAP BW as DWH Best Practice
LSA Implementation
33
44
22
Agenda PDEBW1
Recap of LSA - LSA Related Topics55
BW Data Model and Data Integration66
Appendix - BI Organization & Governance77© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 2
BI Maturity & Data Logistic
BI is the Top Priority of CIOs1.I1.I
11
Agenda PDEBW1
Data Logistic Types and BI Maturity - Overview1.II1.II
Simple Data Logistics – Data Mart Based1.III1.III
Advanced Data Logistics – Data Warehouse Based1.IV1.IV
BI Data Logistic Excellence – Hinderers & Enablers1.V1.V
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 3
The CIOs View and Priorities
InformationExcellence
ProcessExcellence
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 4
BI & Enterprise Application Excellence driveBI Data Logistic Excellence
Information Excellence(Prio 1: BI Applications)
Process Excellence(Prio 2/4: Enterprise Applications)
DataLogistic
BI
No BI-, CPM- andPlanning Excellence
without trustable,accurate, consistent Data
standardization &consolidation of BI data
logistic (EDW)
Standardization andconsolidation of EnterpriseApplications (processes)
standardization &consolidation of BI data
logistic (EDW)
BIData
Logistic
Simple Truth:Simple Truth:No BI without DataNo BI without Data
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 5
A Valid BI Data Logistic asPrerequisite for BI Value
BI BusinessValue Drivers
SAP ERP SAP CRM Other Legacy
Data-Infrastructure, Data-Logistic
BI Applications and FunctionalityStandard Reporting, Agile BI
Corporate Performance Management
BI Framework
ConsistencyFlexibility
EfficiencySuitability
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 6
BI Maturity & Data Logistic
BI is the Top Priority of CIOs1.I1.I
11
Agenda PDEBW1
Data Logistic Types and BI Maturity - Overview1.II1.II
Simple Data Logistics – Data Mart Based1.III1.III
Advanced Data Logistics – Data Warehouse Based1.IV1.IV
BI Data Logistic Excellence – Hinderers & Enablers1.V1.V
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 7
TDWI – Accelerating BI MaturityThe Chasm on The Way to The EDW Shore
Figure 1. The TDWI Maturity Model shows how many organizations evolve their BI solutions.The Gulf and Chasm represent places where many companies get stuck; the bell shaped curverepresents the number of companies in each stage; and the labels above the bell curverepresent the data structure dimension in the model.
Business ValueIntegration
Consolidation
3. Child 4. Teenager 5. Adult 6. Sage
GULF“ ManagementReporting ”
”
“ DataMarts ”
“ DataWarehouses ”
“DW ”
“ BI Services ”
1. Prenatal 2. Infant
GULFCHASM
“ManagementReporting”
“Spreadmarts”
“DataMarts”
“DataWarehouses”
“EnterpriseDW”
“BI Services”
Moving towards a centralized EDW based architecture means we haveto overcome profound challenges (chasm):Non-IT ones like politics, missing strategy and sponsorship andIT ones .....
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 8
Correlation of BI Value & Data Logistics ExcellenceThe Need of Centralization
OperationalReporting
Spreadmarts Data MartsData
WarehousesEnterprise
DWArchitecture
FinancialSystem
ExecutiveSystem
AnalyticalSystem
MonitoringSystem
StrategicSystem
BusinessService
Type ofSystem
“ Drive theBusiness ”
“ Drive theMarket ”
“ MonitorPerformance ”
“ EmpowerWorkers ”
“ InformExecutives ”
“ CostCenter ”Executive
Perception
Value
CostROI
Infant Child AdultTeenager Sage
Static Reports Spreadsheets OLAP/Ad hoc Reports
Dashboards/Scorecards
AnalyticalTools
PredictiveAnalytics
Customer BIEmbedded BI
Prenatal
ROI
AnalyticServicesOperational
ReportingSpreadmarts Data Marts
DataWarehouses
EnterpriseDW
Architecture
FinancialSystem
ExecutiveSystem
AnalyticalSystem
MonitoringSystem
StrategicSystem
BusinessService
Type ofSystem
“ Drive theBusiness ”
“ Drive theMarket ”
“ MonitorPerformance ”
“ EmpowerWorkers ”
“ InformExecutives ”
“ CostCenter ”Executive
Perception
“ Drive theBusiness ”
“ Drive theMarket ”
“ MonitorPerformance ”
“ EmpowerWorkers ”
“ InformExecutives ”
“ CostCenter ”Executive
Perception
Infant Child AdultTeenager Sage
Static Reports Spreadsheets OLAP/Ad hoc Reports
Dashboards/Scorecards
AnalyticalTools
Predictive
ROI
Beyond the Basics: Accelerating BI MaturityWayne W. EckersonDirector, TDWI ResearchThe Data Warehousing Institute 2007
Best practice: centralize before you federate‘...interestingly, organizations that try to federate development andadministration before they have centralized these tasks often havelimited success.’
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 9
BI Maturity & Data Logistic
BI is the Top Priority of CIOs1.I1.I
11
Agenda PDEBW1
Data Logistic Types and BI Maturity - Overview1.II1.II
Simple Data Logistics – Data Mart Based1.III1.III
Advanced Data Logistics – Data Warehouse Based1.IV1.IV
BI Data Logistic Excellence – Hinderers & Enablers1.V1.V
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 10
BI Data-Logistic Types‘Standalone’ Data Mart Architecture
Continental Europe:Sources Data
Marts
Business ValueSemantic IntegrationData Consolidation
1. Prenatal 2. Infant 3. Child 4. Teenager 5. Adult 6. Sage
GULF CHASM“ManagementReporting”
“Spreadmarts”
“DataMarts”
“DataWarehouses”
“EnterpriseDW”
“BI Services”
Business ValueSemantic IntegrationData Consolidation
1. Prenatal 2. Infant 3. Child 4. Teenager 5. Adult 6. Sage
GULF CHASM“ManagementReporting”
“Spreadmarts”
“DataMarts”
“DataWarehouses”
“EnterpriseDW”
“BI Services”“Management
Reporting”
“Spreadmarts”
“DataMarts”
“DataWarehouses”
“EnterpriseDW”
“BI Services”
A definition:
An implementation of an analyticsapplication serving a singledepartment, subject area, or limitedpart of the organization. Usuallyrefers to a physical platform onwhich summarized data is stored fordecision support.
Intra-Departmental:Complexity &Misalignment
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 11
‘Standalone’ Data Marts
Sales
Standalone Data Marts are independently built BI solutions oftenbased on different BI tools.
Different data models i.e. inconsistent semantics & valuesDifferent data management for Data Mart and different BI toolsInhomogeneous, multiple extractions & staging
High departmental acceptance but islands (silos, stovepipes)
HRMaterialManagement
Sources
StandaloneData Marts#
Employee
Material
#
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 12
BI Data-Logistic Types – Flier :Standalone Data Marts
Standalone Data Marts = Departmental BI1. Organization
Departmental, High daily involvement by local Business AnalystLimited IT support
2. Architecture, Reach of Data LogisticInhomogeneous, multiple extractions & stagingData Mart tools have often own data management, multi-dimensional, sql-generator, mdxDepartmental reach
3. Data ModelNo integration between Data Marts,Different data models i.e. inconsistent semantics & values
4. IssuesCross-departmental view: misalignment, islands, communication confusion, high costs,not pursuable, ...
5. AdvantagesDepartmental view: flexible, autonomy
6. Adequateness for Customer typesAdequate only from departmental perspective or for small customer (< 100) with ITknowledge
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 13
BI Maturity & Data Logistic
BI is the Top Priority of CIOs1.I1.I
11
Agenda PDEBW1
Data Logistic Types and BI Maturity - Overview1.II1.II
Simple Data Logistics – Data Mart Based1.III1.III
Advanced Data Logistics – Data Warehouse Based1.IV1.IV
BI Data Logistic Excellence – Hinderers & Enablers1.V1.V
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 14
BI Data-Logistic TypesData Warehouses
BI tools work on Data Marts or more general on special prepared sets of data
Because of the missing integration (islands) between Data Marts, DataWarehouses are established as a common integrated data foundation for DataMarts.
To show the new quality we call these kind of Data Marts ‘Architected’
Business ValueSemantic IntegrationData Consolidation
3. Child 4. Teenager 5. Adult 6. Sage
GULF“ ManagementReporting ”
”
“ DataMarts”
“ DataWarehouses ”
“DW”
“ BI Services ”
1. Prenatal 2. Infant
GULF CHASM“ManagementReporting”
“Spreadmarts”
“DataMarts”
“DataWarehouses”
“EnterpriseDW”
“BI Services”
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 15
BI Data-Logistic TypesData Warehouse Architecture
Sources DataMarts
DataWarehouse
Org
Uni
t A
StagingArea
Bill Inmon - The Data Warehouse‘The Data Warehouse is
a subject-oriented, integrated, time-variant, non-volatilecollection of data used to support the strategic decision-making process forthe enterprise. It is the central point of data integration for businessintelligence and is the source of data for the data marts, delivering a commonview of enterprise data.’
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 16
A Data Warehouse as Integrated Foundationfor Data Marts – Architected Data Marts
MaterialManagement
Sales HR
Material
=Employee
=
Staging Area
Sources
employeematerial
customervendor
Data Warehouse
ArchitectedData Marts
Before Data Marts are loaded the data are integrated to common semanticsgiven by the dwh data model and the values are harmonized.The integrated results are stored persistently and build the data warehouse.Data Marts are filled from the Data Warehouse.
A Data Warehouse is a necessary condition for consistent BI.
Data Warehouse:Standardized extraction & loadIntegrate data with respect tointegrated semantics & valuesStore Integrated resultsLoad integrated data into DataMarts (Architected Data Marts)From this perspective thedata warehouse represents alayer in the informationsystem architecture
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 17
Bill Inmon’s Corporate Information Factory &The Data Warehouse
Copyright ©1999 by William H. Inmon
Conceptual DetailsSubject-orientedIntegratedhistorical completecomprehensiveapplication neutralgranularHigher organizational-unit ownednon-volatile…
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 18
Data Warehouse PrinciplesInformation Completeness, Reusability, Granularity
Customer Dimension Material DimensionA 4712
4713
Customer Material TimeAmount
CompanyCurrency
A 4712 200211 100A 4713 200211 150
Time Dimension200211
Customer Time DocNo Pos MaterialAmount
LocalCurrency
AmountCompanyCurrency
A 20021107-10am 1 10 4711 100 50A 20021107-3pm 1 10 4712 200 100A 20021107-4pm 2 10 4713 300 150
Customer Time DocNo Pos MaterialAmount
LocalCurrency
A 20021107-10am 1 10 4711 100 New bookingA 20021107-3pm 1 10 4712 200 Correction bookingA 20021107-4pm 2 10 4713 300 New booking
Sourcesystem
BW Data Warehouse Layer
daily
weekly dailymonthly
other InfoCubeother InfoCube
BW Architected Data Mart Layer
InfoCube
granularity : week granularity : daygranularity : month
Example:
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 19
DWH & Architected Data Mart Layer
Introducing a Data Warehouse Layer the architecture has two main Layer:
DWH Layerno business driven transformations just
cleansing, unification and integrationtransformations
Based on common definitions
Architected Data Mart Layerbusiness driven transformationsbusiness definedBWA area
BW
DW
HA
rchi
tect
edD
ata
Mar
ts
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 20
Bill Inmon ‘What is a Data Warehouse?’Data warehouses are significantly different from data marts.
From ‘Data Mart Does Not Equal Data Warehouse’, Bill Inmon, DM Direct, November 1999
Integrated, subject-orientedThe data warehouse data is integrated from the many legacy sources.Data warehouses are arranged around the corporate subject areas found in thecorporate data model.
Corporate owned or larger part of the organizationThe data warehouse represents a truly corporate/ higher organizational-unit effort.Usually the data warehouse is built and owned by centrally coordinated organizations
time-variant = complete history, non-volatile = permanent
GranularThe data warehouse contains the most granular data the corporation has. Data martdata is usually much less granular than data warehouse data. The volume of data foundin the data warehouse is significantly different from the data found in the data mart.
Application neutral, non-flavoredThe structure and the content of the data in the data warehouse do not reflect the bias ofany particular department, but represent the corporation's needs for data.
ComprehensiveThe data warehouse is broad in scope and keeps more data than actual requested bythe business/ data marts
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 21
BI Data-Logistic Types – Flier :Data Warehouse
Data Warehouse & Architected Data Marts: Integrated, Cross-organizational BI(on DWH definition level)
1. OrganizationIT managed sometimes BI Competency Center (BI CC)
2. Architecture, Reach of Data LogisticCommon extraction & staging techniques (ETL) for all cross departmental Data MartsData Integration results have own persistenceData Marts served by the data warehouse (architected, reliable data marts)Cross organizational on upper Org-unit
3. Data ModelCross Subject-area integrated
4. Issues, ChallengesBusiness-user view: Often IT-driven, missing business acceptance, needs business buy-in, time to market
5. AdvantagesData warehouse owner view: Integrated reporting, standards, consistency, less TCO thanpure Data Mart architectures
6. Adequateness for Customer typesFor independent operating higher organizational units
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 22
BI Data-Logistic TypesEnterprise Data Warehouse (EDW)
Business ValueSemantic IntegrationData Consolidation
3. Child 4. Teenager 5. Adult 6. Sage
GULF“ ManagementReporting ”
”
“ DataMarts ”
“ DataWarehouses ”
“DW ”
“ BI Services ”
1. Prenatal 2. Infant
GULF CHASM“ManagementReporting”
“Spreadmarts”
“DataMarts”
“DataWarehouses”
“EnterpriseDW”
“BI Services”
Data Warehouses cover a certain part of an organization (country, businessunit, process e.g. FI), which means for larger companies there exist multipleData Warehouses.
Thus from a higher level point of view they suffer from similar issues likestandalone Data Marts. The Data Warehouses are often isolated.
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 23
The Multiple Data Warehouse Landscape Issue:Justification of an Enterprise Data Warehouse
Source
Source
Source
Source
Source
Source
Source Source
LocalSAP BW
LocalSAP BW
Source
Source
SourceLocal
SAP BWUnit 2
SAP BW
LocalSAP BW
LocalSAP BW
Stream BDWH
Stream CSAP BW
GroupSAP BW Stream A
SAP BW
Unit 1DWH
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 24
BI Data-Logistic TypesImpacts of Multiple Data Warehouses
Org
Uni
t A
Org
Uni
t B
Org
Uni
t C
Often we find multiple DWHsin an Organization
Complexity &Misalignment
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 25
Bill Inmon’s Corporate Information Factory &Enterprise Data Warehouse (EDW)
Copyright ©1999 by William H. Inmon
Enterprise Data Warehouse (EDW):A single instantiation of a datawarehouse layer for the entirecorporation or big parts of theorganization is often called theEnterprise Data Warehouse
EDW-Keywordsoffer a ‘single version of truth’extract once & deploy manysupport the ‘unknown’
re-buildnew-build
controlled redundancyprovide a corporate memory
Conceptual DetailsIntegratedhistorical completecomprehensiveapplication neutralgranularcorporate ownednon-volatile…
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 26
Treating Information as Corporate Asset -Flexibility is Key of Growth & Survival
The KnownDon’t limit yourself by just focusing on
current requirements
The UnknownBe prepared for
unpredictable future needs
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 27
The Whole Picture – The EDW
SAP BW @Henkel
Peter Hinzmann, CIO of Henkel, summarizes the motivation:
“We are increasingly focusing on the whole picture, which in turn raises theimportance of centralized management of the company. This process is activelysupported by IT.”
Werner Böttiger, IT project manager at Henkel’s business intelligence competencecenter:
“The enterprise data warehouse (EDW) layer does lead to a degree of redundantdata, but this can be kept under control.
All transactional data from the source systems need to pass the EDW layer before itis written to the final data destinations or data marts.
This enables us to safeguard data quality and create a single version of truth thatcontains all historical data,”
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 28
BI Data-Logistic Types – Flier :EDW
EDW & Architected Data Marts: Integrated, Corporate BI
1. OrganizationBI Competency Center – BI CC (IT, analysts, business)
2. Architecture, Reach of Data LogisticLike DWH but for entire Corporation (may be division)Single Point of Truth, historical complete, comprehensive, granular, integrated,Architected Data Marts built on EDW
3. Data ModelCross Company
4. Issues, ChallengesIT as driver, missing C-level sponsorshipMissing business acceptance, needs business buy-in, time to market
5. AdvantagesBI on entire value chain,Integration, standardization, new information culture, competitive advantagesEnabler for consistent ‘BI as service’
6. Adequateness for Customer typesFor all customers within highly competitive markets, High volume, Global organization
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 29
Ralph Kimball’s view on Data WarehouseLayer
Figure 1.1 The basic elements of the data warehouse.
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 30
Meta Group I
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 31
Bill Inmon's Corporate Information Factory andThe Enterprise Data Warehouse
DSS ApplicationsDepartmental Data Marts
EDW
MarketingAcctg Finance
Sales ERPERP
ERP
CRM
eComm.
Bus. Int.
ETL
GlobalODS
Oper.Mart
Explorationwarehouse/data mining
* Source: Bill Inmon
Stag
ing
Area
localODS
DialogueManager
CookieCognition
Preformatteddialogues
Cross mediaStorage Management
Near lineStorage
Web Logs
SessionAnalysis
Internet
ERPCorporate
Applications
ChangedData
GranularityManager
Archives
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 32
BI Maturity & Data Logistic
BI is the Top Priority of CIOs1.I1.I
11
Agenda PDEBW1
Data Logistic Types and BI Maturity - Overview1.II1.II
Simple Data Logistics – Data Mart Based1.III1.III
Advanced Data Logistics – Data Warehouse Based1.IV1.IV
BI Data Logistic Excellence – Hinderers & Enablers1.V1.V
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 33
Why are there Chasms & Gulfs ?
Business ValueSemantic IntegrationData Consolidation
3. Child 4. Teenager 5. Adult 6. Sage
GULF“ ManagementReporting ”
”
“ DataMarts”
“ DataWarehouses ”
“DW”
“ BI Services ”
1. Prenatal 2. Infant
GULF CHASM“ManagementReporting”
“Spreadmarts”
“DataMarts”
“DataWarehouses”
“EnterpriseDW”
“BI Services”
zero high
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 34
Different Perspectives on BIBusiness & IT
Business perspective on Informationsolutionsolution, business driven BIbusiness driven BI
Flexible, time to marketAutonomy, agilityPerformanceConsistencyHigh availability
IT perspective on Informationdata, technology driven BIdata, technology driven BI
Manage BI (Development, Maintenance,Operation, Administration, Housekeeping)Lower TCO, TCD (Total Cost ofOwnership, Development)Reduce number of BI tools (BIConsolidation)
BI BusinessValue Drivers
SAP ERP SAP CRM Other Legacy
Data-Infrastructure, Data-Logistic
BI Applications and FunctionalityStandard Reporting, Agile BI
Corporate Performance Management
BI Framework
ConsistencyFlexibility
EfficiencySuitability
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 35
The preferred BI flavor/ tool differs fromemployee to employee, from departmentto department, from organizational-unit toorganizational-unit
Offering different, adequate BIfunctionalities to different business-usertypes is fine but often results in anuncontrolled bunch of BI-tools, whichmeans per se in high costs
But this is not the main issue:A dispersed BI landscape implies as
experiences show an island like,dispersed data infrastructure.
This results in even higher costs butmore important doubts the value of BI
Business Perspective on BIOrganizational Egoism
Group
Division A Division B Division C Division D
......BUUnit ....... .......
Sales FI HR
SpainUK
FranceNordic Different BI tools and
Data infrastructures
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 36
SAP AG 2007, SAP Skills 2007 Conference / B2 / 37
Local Business and Issues of Dispersed BI DataLogistics
Source
Source
Source
Source
Source
Source
Source Source
LocalSAP BW
LocalSAP BW
Source
Source
SourceLocal
SAP BWUnit 2
SAP BW
Organizational/ business and also IT egoisms cause dispersedBI data-logistics. They often hinder any evolutionary step
towards an architectured BI data-logistic.
LocalSAP BW
LocalSAP BW
Stream BDWH
Stream CSAP BW
GroupSAP BW Stream A
SAP BW
Unit 1DWH
Dispersed BI Data Logistic:Missing Service level definition
– Uncontrolled data flows– Uncontrolled extractions
Redundant developmentInconsistent data model, KPIsNo overall BI / DWH Strategy
This results in:Silos, IslandsUnreliable Information-basisUncontrolled redundancyHigh Costs
Dispersed BI Data Logistic:Missing Service level definition
– Uncontrolled data flows– Uncontrolled extractions
Redundant developmentInconsistent data model, KPIsNo overall BI / DWH Strategy
This results in:Silos, IslandsUnreliable Information-basisUncontrolled redundancyHigh Costs
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 37
Why are there Chasms & Gulfs ?Non-IT Reasons
Mature BI Data logistics are hindered by local interests:
If there is no organizational momentum toward a common goal, then the bestarchitecture, the best framework in the world is bound to fail.
W.H. Inmon“
But any advanced BI data logistic like an Enterprise Data Warehouse depends onestablishing a high degree of central governance. This again depends on a
Clear commitment of c- (orporate) management (sponsorship) – not CIO
Particular, Local interests overrule Group, Corporate interests
Delivering consistent information across the business depends on shareddefinitions and business rules for collecting, processing and delivering it.Any significant improvement in information – in terms of
quality, consistency, timeliness and accuracy– depends on gaining control of information processes and definitions.
a customer architecture council
“On the other hand side we find the awareness:
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 38
Is IT Ready for EDW?IT and Issues of BI Data Logistics
DW
H
DW
H
DW
H
data
mar
ts
Staging
gene
rate
dem
and
Source
Extraction
dem
and
supp
ly
Staging
Source
Extraction
volume
...
Cop
a
Staging
Source
Extraction
no of applications
We miss:Service level definitionsGuidelines & Best PracticesQuality-control in DevelopmentComprehensivenessCompleteness....
This results in:Increasing Complexity
– Increasing operational cost– Increasing maintenance costs
Low flexibility to answer ‘the unknown’– Increasing time-to-market
Consistency issues
We miss:Service level definitionsGuidelines & Best PracticesQuality-control in DevelopmentComprehensivenessCompleteness....
This results in:Increasing Complexity
– Increasing operational cost– Increasing maintenance costs
Low flexibility to answer ‘the unknown’– Increasing time-to-market
Consistency issues
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 39
Why are there Chasms & Gulfs ?Is IT Ready For EDW ?
BI is totally underestimated and misunderstood by IT! (and others)
BI is highly different than dealing with operative requirements.BI is daily business-related (at all org-levels) and thus highly volatile/ dynamic
BI data-logistics must be highly flexibleBI deals with historical data what creates high data volume growth
BI data-logistics have to manage often 10 times or more of dataBI never endsBI is implemented incrementally, BI-application after BI-application,
Data volume growth but just so important,Growth of meta-data
BI means steadily increasing maintenance-, administration- and operation-workloadBI data-logistics have to be highly transparent, standardized robust and automatedBI is steadily under time-to-market pressure
BI data-logistics cannot rely on today. They must ‘support the unknown’ the unforeseeable.
BI deals with data, operative-business with processes!As BI is underestimated, the EDW is underestimated!This is the best prerequisite to fail!
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 40
Crossing BI Data Logistic Gulfs & Chasms –Is IT Ready for a BI Excellence Framework? *
BI BusinessValue Drivers
SAP ERP SAP CRM Other Legacy
Infrastructure, Data-LogisticEnterprise Data Warehouse Layered, Scalable
Architecture
Applications and FunctionalityStandard Reporting, Agile BI
Corporate Performance Management
Organization Process
StrategyInformation as Corporate Asset
MethodologyBI Competency Centre
BI Framework*
ConsistencyFlexibility
EfficiencySuitability
Realization
Alignment
* BI Framework introduced by Gartner
BI Excellence is more than Infrastructure/ Data-Logistic Excellence:
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 41
Treat Information as Corporate Asset -Mature BI Data-Logistics (EDW)
Increasing awareness about importance of a proper SAP BW data-logistic for Data andMeta Data
Customer investments for an adequate SAP BW infrastructure to manage data and metadata consistently and in a comprehensive manner
Activities run under the title ‘Enterprise Data Warehouse’Most common to all activities is the concept of a layered architecture of information systemsintroduced by Bill Inmon The Corporate Information Factory (CIF)
SAP BWLayered, Scalable Architecture
(LSA)
Com
plex
ity &
Cos
t:A
dmin
istr
atio
n,M
aint
enan
ce
No of Application Scenarios (Data Marts)/ Volume
SAP BWwith no corporate standards
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 42
BW Layered, Scalable Architecture (LSA) for EDW
BI Maturity & Data Logistic11
Model-driven SAP BW as DWH Best Practice
LSA Implementation
33
44
22
Agenda PDEBW1
Recap of LSA - LSA Related Topics55
BW Data Model and Data Integration66
Appendix - BI Organization & Governance77© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 43
BI Data-Logistic & Applications Modeling withSAP BW
Mod
elin
g A
ctio
nsO
perate/ Adm
inistrationBusiness requirements
DWH data model
Multi dimensional model
Physical Star Schema
Logical DWH & Staging
Physical DWH & Staging
Data Transfer,Transformation, Error-handling
Locate source &map data & model, extractor
Erwin, Visio, Eclipse (planned)
BCT BW Reference Data Model
BW InfoCube/ MultiCube
Star Schema/ BWA index
BW DSOs
Generated PSA, DSOs tables
InfoPackage, DTPs, DAPsRule-editor, Error-DTP
BCT DWH Data Model,Extractors: BCT, Generic, UDC
Process Design
Deploy BI applications
Monitor BI applications
Performance, Tuning
Run, Recover
Housekeeping
BW Process Chains
BW Transport
BW Monitoring & Statistics
BW Aggregates, BIA, NLS
BW Requests/ Process Chains
BW Housekeeping Proc Chains
BW models and implement all relevant BIdata-logistic areas. Data Warehousing withBW is end to end model-driven.This is a prerequisite for designing datawarehouses in a transparent, standardized,robust and best practice way.
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 44
Non-SAP-Sourcedata model
BW Model-Driven Data Warehousing based onBCT Data Warehouse Reference Data Model
Buildingblocks DWH Stores:
DTPs, DSOs
ArchitectedData Marts:InfoCubes/ BWA
SAP-Sourcedata model
Non-SAP Source BSAP Source A
BW Model-driven DWH:1. DWH data modeling
Reference data modelInfoObjects + relations
2. DWH structure modelingInfoProviders
3. DWH operations modelingExtract, load, transformTransfer, Error-handlingAdmin, Monitor
ETL/ StagingPSA, InfoPackagesmap models:
provided by BW
extractor
map models:user-defined
extractor
Subject-areas
BW is from all perspectives fully model-driven
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 45
SAP BW Model-Driven Data WarehousingCornerstones
Designing a Data-Logistic with SAP BW is from all perspectives fully model-driven:
The Reference Data Model (BCT)Provides mapping of SAP-source data model to BW reference data model (InfoObjects)For all Non-SAP sources
Customer does not start on a ‘blank’ sheet of paper (-> reference model)Mapping of source data model to reference data model by customerFlexible, expandable
The Meta-objects to design persistent & virtual elements of a data-logisticPersistent Data-containers (InfoProviders: InfoCube, InfoObject, DSO, PSA)Access Virtualization (InfoProviders: MultiProvider, Virtual InfoProvider, InfoSet,..)Staging Virtualization (DataSource, InfoSource)
The Meta-objects to design the operations between persistent elementsExtract, Transport, Load (InfoPackage)Transformation (Transformation Rules)Error-HandlingTransfers (DTPs)Complex data movements (Process chains)
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 46
Conformed dimensions enable virtual SAP BW MultiProvider scenarios
FACT Table
Material_Dimension_ID..........._Dimension_ID.............Dimension_ID.............Dimension_ID
Key figure 1Key figure 2InfoCube
B
Local Part ofMaterial Dimension
Material_Dimension_ID
0MATERIAL0PROD_HIERMaterial Dimension
Table
FACT Table
Material_Dimension_IDSalesOrg_Dimension_IDTime_Dimension_IDCustomer_Dimension_ID
Sales AmountQuantity InfoCube
A
Material_Dimension_ID
0MATERIALMaterial Dimension
Table
Gebiet 1 Geb iet 2 Gebiet 3
Bezirk 1
Gebiet 3a
Be zi rk 2
Regio n 1
Gebiet 4 Gebiet 5
Bezirk 3
Regi on 2
Geb iet 6
Bezirk 4
Gebie t 7 Geb iet 8
Bezirk 5
R egion 3
Vertriebsorg anisation
Material HierarchyTable
0MATERIALLanguage CodeMaterial Name
Material Text Table
Material Master Table
0MATERIALMaterial Type
Local Part ofMaterial Dimension
Conformed, SharedPart of
Material Dimension
Material Dimension:local & conformed part
BW Data Warehouse & Architected DMs:Conformed Dimensions - Preventing Silos
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 47
The SAP BW MultiProvider Concept
End-UserReporting & Analysis need
logical (dimensional) model
SAP BW InfoCube(s),DS-Objectphysical implementation
basic factsnot report specific
SAP BW MultiProvidervirtual Structures
basic factsnot report specific
optional
SAP BW BEx Queryvirtual Structures
complex KPIsnavigation behaviorreport (type) specific
SAP BW BEx Reportend-user requirement
specific
SAP BW
Concept of SAP BW MultiProvider
MultiProvider separate physical persistent SAP BW Structuresfrom Queries and Query dependent BEx-objects
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 48
BW Data Warehouse & Architected DMs:MultiProvider and Flexibility
+C1C1
Expand the scenario(C1) seamlessintroducing a newscenarioFlexibilitySave investments
C1C1
ManagingWorkload (logicalpartitions)PerformanceNew contentFlexibilityRebuildFlexibility
SAP BW Queries
virtualSAP BW MultiProvider
Separate optimalphysical DB Schemafrom logical SchemaPerformance
C1C1persistentSAP BW Structures
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 49
BW Data Warehouse & Architected DMs:Data Marts Built on BCT Reference Model
MaterialManagement
Sales HROLTP
Material
= =Employee
Staging Area:BW PSA
Load:BW InfoPackage
ArchitectedData Marts:
InfoCubes, MPs
Transfer:DTPs, Open Hub
BW
Achieving Data Mart harmonization applying the BW reference data modelNot necessarily explicit data warehouse persistency
TransformationRules
Non-SAP Extraction:BW Extractors,BW DBConnect,
BW Generic,FlatFile,
Univ.DataConnect
BW
Data-Logistic M
odeling
SAP
BW
Ref
eren
ce D
ata
Mod
el
0EMPLOYEE0MATERIAL
0CUSTOMER0VENDOR
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 50
BW Data Warehouse & Architected DMs:Limitations
The ‘just use’ SAP BW DWHfocused on InfoCubes,
InfoObject master datamore or less summarizedsupport the known
SAP BWs are built on today's end-user requirementsDesigned to answer the knownAnswering the unknown is limited by
Granularity of InfoCubes and DS-Objects (data content)– (historical completeness)
Business-scope driven extraction– (extracted fields – process comprehensiveness)
Business rules applied to original extracted dataAvailability of master data history
Limited Completeness & comprehensivenessthus overall flexibility is limited to react on time
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 51
BW Data Warehouse & Architected DMs:Summary
+ Usage of BCT reference data modelCommon model-based consistent Data Marts
+ Relying on the power of the BW model-driven approachEnables implementation of even most complex BI scenarios
Customers who have no explicit or corporate BI/ DWH strategy except ‘use SAPBW’ - Little awareness about the meaning of incremental BI
limited scalability, increasing TCO when BW grows
Solution focus on InfoCubes/ reporting - usage of intermediate storage (DSOs)only if required to cover concrete business scope
Limited flexibility
Little standards & guidelines, which exceeds the generic BW offering
We find the ‘just use SAP BW approach’ with a lot of customers. Theapproach works well until the BW-system reaches a certain size in terms ofdata-volume and number of BI applications. Then the TCO will increase andscalability, maintenance, performance and operational issues will likely occur.For corporate BI data warehouse architectures just relying on BWs genericdata warehouse approach is not sufficient!
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 52
Implementing an EDW-Based Architecture inSAP BW
I definitely need to gofor an EDW-based
layered architecturefor SAP BW!
But how to do it ?
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 53
BW Layered Scalable Architecture (LSA) for EDW
BI Maturity & Data Logistic11
Model-driven SAP BW as DWH Best Practice
LSA Implementation
33
44
22
Agenda PDEBW1
Recap of LSA - LSA Related Topics55
BW Data Model and Data Integration66
Appendix - BI Organization & Governance77© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 54
The Layered, Scalable Architecture (LSA) for EDW
Reference LSA Overview3.I3.I
33
Agenda PDEBW1
LSA Building Blocks – Data Layer3.III3.III
LSA Building Blocks – Data Domains3.V3.V
LSA Building Blocks – Data Model3.VI3.VI
LSA Building Blocks – Layer & Master Data3.IV3.IV
Reference LSA & Customer LSA3.II3.II
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 55
Everything Seems to be Clear, Doesn‘t It?
• EDW &Layering• Chasms
?
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /20090210/ Page 56
The Broad View of BW on EDW
Traditional EDW SAP BW EDW
Companiessituation
operational world fragmentationno view across the business
Increasing operational worldharmonization
Yet limited view across the businessScope-Areas
Cross-/Corporate BI
In scopelittle information freshness (monthly)
In scopehigh information freshness (daily)high flexibility
Local- /depart. BI
Not in scope In scopestandard reporting with local adoptionsflexible roll out
Operational BI Not in scope Partly in scope
Evaluation Long time for ROIHigh Latency (e.g. monthly)High costsLow synergies for local/
departmental BIOverall acceptance problems
Incremental approach to EDWImmediate ROI (local BI)Std. DWH Latency (day) to Low (RDA)Standardization lowers costsHigh synergies for all BI flavorsIncreasing acceptance over time
Note: of course we find also the ‘traditional’ EDW approach using BW!© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 57
Crossing the EDW Chasm:The BW Layered Scalable Architecture
SAP Customer Experiences & Feedback :
Nestlé
Kraft Foods
Arla Foods
Adidas
Edeka
Beiersdorf
HenkelJapan
TobaccoPhilips
Samsung
Novartis
Syngenta
BASF
LandHessen
ShellDownstream
Mc Kesson
Statoil
Best Practices &Best Practices &EDW PrinciplesEDW Principles
BW Layered Scalable Architecture (LSA) –The Reference Architecturefor BW on a Corporate Scale
LSA
Nike
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 58
The Layered, Scalable Architecture (LSA) for EDW
Reference LSA Overview3.I3.I
33
Agenda PDEBW1
LSA Building Blocks – Data Layer3.III3.III
LSA Building Blocks – Data Domains3.V3.V
LSA Building Blocks – Data Model3.VI3.VI
LSA Building Blocks – Layer & Master Data3.IV3.IV
Reference LSA & Customer LSA3.II3.II
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 59
The BW Layered Scalable Architecture (LSA)describes the design of
service-level oriented, scalable, best practice BWarchitectures founded on accepted EDW principles*.
The BW LSA serves as a reference architectureto design transparent, complete, comprehensive
customer DWH architectures (Customer LSA).
The Customer LSA describes corporate standardsto build BI applications in a
performant, maintainable, flexible manner.
BW Layered Scalable Architecture (LSA)
* As introduced in Bill Inmon‘s Corporate Information Factory (CIF)© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /20090210/ Page 60
BW LSA: The Reference Architecture
LSABuilding Blocks
Reference
Describescore structures &
definitions
LSAImplementation
Reference
Describesdesign standards tobuild BI applicationsfounded on building
blocks
LSAOperationsReference
DescribesSupport
Scenarios
Meta Data ManagementNaming ConventionsOrganization (InfoAreas)
Life CyclesInformationMeta ObjectLSA
AdministrationData BaseHousekeepingMonitoring
Transport
Security
Data Staging/ Management fortransactional & master data
Persistent ObjectsFlows - scheduled/ recoveryTransformationProgramming (Abap)Organization (Process Ch.)
BW LSA
Landmark Building BlocksLayerDomainsData Model &
Data IntegrationAssistant Building Blocks
Data QualityLandscapeETLStorageOwnership/ OrganizeDevelopment concept
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /20090210/ Page 61
LSA Building Blocks –The Architecture Cornerstones
The LSA Building Blocks
are the cornerstones of your futurearchitecture thus having a decisive influenceon the overall success of your future BW EDW
describe the general BW EDW layoutindependent from concrete BI applications andBI projects.
Landmark Building Blocks deal with architectureareas that need fundamental decisions beforedoing implementations – they play the same rolelike the supporting structures (pillars & ceilings) ofbuildings.
Assistant Building Blocks deal with architectureareas that are normally less prior with respect tothe point in time you should decide and roll outcorporate standards.
LSABuilding Blocks
Reference
Describescore structures &
definitions
Landmark Building BlocksLayerDomainsData Model &
Data IntegrationAssistant Building Blocks
Data QualityLandscapeETLStorageOwnership/ OrganizeDevelopment concept
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 62
LSA Landmark Building BlocksLayer & Domains
There are two areas to standardize the architecture of persistent data stores:
1. Structure the data stores in a flow from PSA to InfoCubes with respect totheir role and the offered services – define Data Layer
2. Structure (split/ collect) the data within the Layer to guarantee Layer andadvanced services – define Data Domains
LSA ArchitecturedNon-Architectured
Domain
Layer
data
flow
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 63
LSA Assistant Building Blocks I
ETL (Extraction, Transformation, Load)Do you expect extensively data from non-SAP sources?
If the answer is ‘yes’, it is meaningful investigating an ETL-tool like SAP Data IntegratorIf SAP systems are the initial and primary BW EDW sources, you just don’t care. May be later
Data QualityDo you have considerable data quality issues?
If the answer is ‘yes’, it makes sense thinking about a Data Quality tool.If integrated SAP systems are the initial and primary BW EDW sources, you don’t care.
Landscapeoften reduced to ‘Do I need a single or a multiple BW instance’ landscape. This topic hasbecome more and more an assistant one, because of the arrival of new technologies and thetransparent support by BW (BW Accelerator, Near Line Storage s. ‘Storage’).The landscape architecture comes into focus
if we have to support mission critical BI or to observe legal restrictions or with other customerspecific requirements
if it comes to agile BI and local autonomy (federated landscapes, SAP Newton appliancesand BW EDW)
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 64
LSA Assistant Building Blocks II
StorageTraditionally all data of a BW DWH are hosted by an RDBMS.This is for large scale BW EDWs not adequate (this applies also to smaller BWs)
Use RDBMS compression if available and usable (e.g. DB2 ‘deep compression’)Rarely used data should be hosted by Near Line Storage (NLS). NLS tools compress your
data up to 95% (e.g. SAND) without destroying your Service Level Agreements (SLAs).The BW Accelerator (BWA) must be part of the overall architecture
Ownership/ Organization
Designing, implementing and operating a BW EDW need strong governance.
A BI Competency Center (BICC) should be established if not existing yet.
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 65
The Layered Scalable Architecture Goals -Standardization, Transparency, Efficiency
architecturedarchitecturednonnon--architecturedarchitectured
Non-Architectured
D a
t a
f l o
w
D a
t a
f l o
w
LSAArchitectured
Large scale BWData Warehouses
(EDWs) shouldfollow architectureprinciples like we
can observeconstructing large
buildings:standardized,scalable, no
squiggles, efficientusage of materials,earth quake save.
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /20090210/ Page 66
DEPLOYING WEBI ON SAPWith large scale BWs there is only one way to be successful:
standardize, standardize, standardize ...in a way making your BW transparent, robust & in the broadest sense
scalable
The BW Layered Scalable Architecture is all about:What, Where, When & How to Standardize
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 67
The Layered, Scalable Architecture (LSA) for EDW
Reference LSA Overview3.I3.I
33
Agenda PDEBW1
LSA Building Blocks – Data Layer3.III3.III
LSA Building Blocks – Data Domains3.V3.V
LSA Building Blocks – Data Model3.VI3.VI
LSA Building Blocks – Layer & Master Data3.IV3.IV
Reference LSA & Customer LSA3.II3.II
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 68
From LSA Reference Architecture toCustomer LSA
LSABuilding Blocks
Reference
LSAImplementation
Reference
LSAOperationsReference
SAP‘s LSAReference
Customer-LSABuildingBlocks
Customer-LSAImplementation
Standards
Customer-LSAOperationsStandards
Customer-LSAStandards
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /20090210/ Page 69
BI-Projects built on Customer LSA Standards
LSABuilding Blocks
Reference
LSAImplementation
Reference
LSAOperationsReference
SAP‘s LSAReference
Customer-LSABuildingBlocks
Customer-LSAImplementation
Standards
Customer-LSAOperationsStandards
Customer-LSAStandards
BI-Project-DesignBased on
Customer LSA
Plan-Build-RunApply: Individuell BI project design
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /20090210/ Page 70
Customer LSA ‘Handbook’Shell
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 71
Customer LSA ‘Handbook’Shell
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 72
Customer LSA ‘Handbook’Land Hessen
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 73
BI-Project-Design
Life Cycle of The Customer LSA
BW LSA: The Reference Architecture
Customer LSA : Standards - Handbook
BI-Project-DesignBI Project Design
Step 3:PerfectPerfect
Customer LSA
Step 4:UpdateUpdate
Customer LSA
Step 1:DesignDesign
Customer LSA
Step 2:ApplyApply
Customer LSA
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /20090210/ Page 74
Different Starting Points –Different Customer LSAs
BW on Corporate/ Divisional Level - BW EDW• Consolidation of multiple BWs into a single often global BW – (consolidation)• Reengineering of a BW getting on in years – (reengineering)• Replacement of multiple legacy DWHs by a single BW (replacement)• Allied set up of a BW with an (global )ERP roll-out (alliance)
BW1
BW2
BW3
BWn
Con
solid
atio
n
Pro
lifer
ated
BW
Ree
ngin
eerin
gDWH1
DWH2
DWH3
DWHn
Rep
lace
men
t
New
Cor
pora
teE
RP
(s)
Allia
nce
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 75
Reference LSA & Customer LSA
As background and targets of each customer differ, the preferred servicesdiffer and thus the Customer LSAs will differCustomers differ with respect to:
State of integration (master data, source systems (processes)Painful experience -> Control & Influence (outsourcing)SkillsOverall governance, sponsorship, commitment.In general expections with respect to:
Availability, error-tolerance, robustness, audits :Protection against human errors, Data flows and persistency's (DSO-
Objects) should be resistible against all kind of errorsReport availability at 8 am (local time), 24x7 operationsFast (SLA) recovery without touching source system
Scalability (application scalability)Fast new-build of BI-applications without accessing Sources
Scalability (performance)Flexibility, Protection of investment
manage seamless changes in OLTP world
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /20090210/ Page 76
The Layered, Scalable Architecture (LSA) for EDW
Reference LSA & Customer LSA3.I3.I
33
Agenda PDEBW1
LSA Building Blocks – Data Layer3.III3.III
LSA Building Blocks – Data Domains3.V3.V
LSA Building Blocks – Data Model3.VI3.VI
LSA Building Blocks – Layer & Master Data3.IV3.IV
Reference LSA Overview3.II3.II
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 77
‘Layering’ means horizontal structuring/ modeling of BWThe data flows are organized in a unified, service-oriented way. Parameters are:
Coverage, comprehensiveness (Process, User demands)GranularityHistory (completeness)Reusability (BI application scalability)Recovery (robustness, availability)QualityIntegration statusAccess-rights (End-user)Life-cycle.....
LSA Landmark Building BlocksData Layer/ Layering of Data
Non-Architectured
D a
t a
f l o
w
D a
t a
f l o
w
Value of Data Layer:+ Highly Transparent & Flexible
+Development, Maintenance+Administration, Operations+Database-Integration
LSAArchitectured
Layer
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 78
Basic Design Aspects of BW LSAEDW & Architected Data Marts Layer
There are two main Layer:
EDW LayerSingle Point of Truthreusable
no business driven transformationsjust cleansing, unification andintegrationtransformations
based on common definitionsnaming
Corporate Memory orCorporate/ Group Data Repository
…
Architected Data Mart Layerbusiness driven transformationsbusiness definedBWA area
BW
EDW
Arc
hite
cted
Dat
a M
arts
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 79
LSA Reference Layer Quick Intro
LSA
Reporting Layer (Architected Data Marts)
Business Transformation Layer
Operational D
ata Store
Data Propagation Layer
Quality & Harmonisation Layer
CorporateMemory
Data Acquisition Layer
Virtualization Layer
EDW layerSingle Point of truthReusableCorporate ownedGranular
ArchitectedData Mart Layer
User
Sources
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 80
LSA Reference Layer Quick Intro
LSA
Reporting Layer (Architected Data Marts)
Business Transformation Layer
Operational D
ata Store
Data Propagation Layer
Quality & Harmonisation Layer
CorporateMemory
Data Acquisition Layer
Virtualization Layer
extractor inbox, 1:1 fromextraction, temporary
source system like service level,comprehensive, complete, master theunknown, long term
create harmonizedview, guaranteequality
EDW LayerSingle Point of truthReusableCorporate ownedGranular
BI Applications/Architected
Data Mart Layer
digestible, ready toconsume, integrated,unified data
apply business logic
reporting, analysis ready abstraction near real time,operational like
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 81
LSA Reference Layer BasicsFrom Acquisition to Reporting Layer
SourceSystems
EDW Layer
Data flow
Corporate Memory
BusinessTransformation
Layer
ReportingLayer
PropagationLayer
AcquisitionLayer
Data Mart LayerHarmonize/
QualityV-
Layer
DM1
DM3
DM2Layering applies to transactional & master data!To master data however in a simpler form
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 82
LSA Reference LayerAcquisition Layer
SourceSystems
EDW Layer
Data flow
Corporate Memory
BusinessTransformation
Layer
ReportingLayer
PropagationLayer
AcquisitionLayer
Data Mart LayerHarmonize/
QualityV-
Layer
DM1
DM3
DM2
Acquisition LayerPer DataSource per source system
DataSource should be comprehensive (offer allrelevant process information)
Fast inbound & outbound to targetsAccept data from extraction with as less aspossible overhead – no early checks, merges,transformations i.e. (1:1)
Stamps the extracted records uniquely forconsistent internal DWH processing and tracking(request handling)
Provides abstraction of DWH from sourcesProvides short term history of extracted data for
immediate / short term data inspection
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 83
LSA Reference LayerData Acquisition Layer Flier
Criteria Characteristics
potential sources Operational systems, other data warehouses, any source
potential targets Data Propagation, Harmonisation & Quality, Corporate Memory, ODS
reusability Yes
transformations None, 1:1; possible enrichment of reusable informationen
granularity single records, granularity of DataSource/ most granular
content DataSource comprehensive: more fields than actually requested by Data-Martse.g. all fields of a Business Content DataSource
history 2-4 weeks, short term storage
main services Decouple OLTP extraction cycle from internal BW data processingRobust & fast inbound and distribution
store & deploy Unmerged data (DataSource & source system specific) – copy of delta queueFast store: PSA Element, (Write Optimized DSO)Fast distribution, for large DWH: Scalability thru parallelism – single thread from
source system, scalable parallelism (logical/ semantical partitioning) toconsumers (targets)
update Request by request; no overwrite/ update of loaded records.
housekeeping Regular content deletion
Impl
emen
t.O
p.D
WH
Ser
vice
s
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /20090210/ Page 84
LSA Reference LayerCorporate Memory Layer I
SourceSystems
EDW Layer
Data flow
Corporate Memory
BusinessTransformation
Layer
ReportingLayer
PropagationLayer
AcquisitionLayer
Data Mart LayerHarmonize/
QualityV-
Layer
DM1
DM3
DM2
Corporate Memory (CM)‘Think about the Corporate Memory Layer as your DWH and BI lifeinsurance‘:Mastering tasks, which are unforeseeable (‘the unknown’) andmanageable only violating SLAs:
Avoiding re-init from source(s)(Disaster-) recoveryEnhancing and new build of Data MartsChange of source master data transformationsReorganizations
the CM layer offers a source system like Service Level:Granular dataComprehensive data (high coverage of underlying process)Complete history of loaded data
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 85
LSA Reference LayerCorporate Memory Layer II
SourceSystems
EDW Layer
Data flow
Corporate Memory
BusinessTransformation
Layer
ReportingLayer
PropagationLayer
AcquisitionLayer
Data Mart LayerHarmonize/
QualityV-
Layer
DM1
DM3
DM2
Corporate Memory (CM)‘Think about the Corporate Memory Layer as your DWH and BI lifeinsurance‘:
Mastering tasks, which are unforeseeable (‘the unknown’) andmanageable only violating SLAs
The CM layer offers a source system like Service Level1:1 from Acquisition as rule of thumbin addition harmonized data may be stored in a dedicated Corporate
Memory Object after complex harmonization/ transformationIf we have the same DataSource offered by multiple source-systems we
should go for a single CM DSO. (Data lifecycle management must exist!)NLS is the right place for CM data
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 86
LSA Reference LayerValue of a Corporate Memory
You can think about the Corporate Memory as the area, which protects your investments, whichprotects you against all kind of surprises. Surprises in an architecture are unwelcome but theywill happen.
you should never rely on what business ask you to deliverthe corporate memory stores more information (fields) about a process than you are actually asked for.In fact we recommend to store all available describing fields of a process. This is easy for SAP sourcesas we extract data using all fields of a delivered DataSource.– Thus the Corporate Memory is comprehensive.we do not aggregate information/ bookings. We store all extracted records.
– Thus the Corporate Memory is historical complete (e.g. for bookings we have the complete historyof a document)
you may believe that you can repeat the loads or do again an initial load if you experienceproblems or new requirements. This is often not realistic or just not possible.
Not realistic:– To repeat historic loads with delta loads– initial loads cause a downtime in the operative system what is often not acceptableNot possible– In the operative system the data will be archived
you should always be aware that when you transform/ map original data ( Quality &Harmonization Layer) the mapping rules may change.
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 87
LSA Reference LayerValue of a Corporate Memory - Summary
The CM is a layer of choiceto guarantee utmost possible flexibility to react on the unknown, the unforeseeableif downtime of operative systems for recovery purposes (new initialization) isunacceptable (24x7) or if archiving takes place in operative system. In mostrecovery situations the Corporate Memory will help.in general for robust and standardized recovery proceedingto keep more information than actually needed or which can actually be integratedwithout impacting daily operations
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 88
© SAP 2008 / Page 89
LSA Reference LayerCorporate Memory Flier
Criteria Characteristics
potential sources Data Acquisition Layer, (Harmonization Layer)
potential targets Data Propagation Layer (Business Transformation Layer, Reporting Layer)
reusability Yes
transformations Often 1:1, technical harmonization (compounding)
granularity All extracted records for delta loads (copy of delta queue)content comprehensive (all fields of a DataSource)
additional fields to simplify administration & access
history complete
main services source system like SLA‘Single Point of Truth’ of extracted data for delta/ changed data for fullultimate area for application recovery, new-build and DWH reorganizationmanage the unknown
store & deploy DataSource specificWrite-optimized (wo) DSO for delta loads, normal DSO + wo DSO for fullnot directly in flow to applications (branched out)
update Request by request; no overwrite/ update of loaded records
Data life cycle Should mainly reside on NLS (Near Line Storage)
DW
H S
ervi
ces
Impl
emen
tO
p.
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /20090210/ Page 89
LSA Reference LayerHarmonization & Quality Layer
SourceSystems
EDW Layer
Data flow
Corporate Memory
BusinessTransformation
Layer
ReportingLayer
PropagationLayer
AcquisitionLayer
Data Mart LayerHarmonize/
QualityV-
Layer
DM1
DM3
DM2
Harmonization & Quality LayerThe services of this layer are data and data modelintegration and quality checked data e.g.
Technical harmonizationformat, lengthsimple content harmonization (e.g. date)integrating local master data to a single model (key
compounding, concatenation)Integration of local master data to group master data
(group data model)Checking Integrity of loaded dataUnification of data: adding e.g. load time, origin..Advanced content cleansing
e.g. detect duplicates, address cleansing etc. -> DataServices
...Harmonizing data and guaranteeing quality of data canmean high efforts or no efforts at all. Customers whoinvested in process integration and common coded/integrated master data will harvest now.
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 90
LSA Reference LayerHarmonisation & Quality Layer Flier
Criteria Characteristics
potential sources Acquisition Layer, CM
potential targets Data Propagation Layer, (occasionally Corporate Memory)
reusability Yes
transformations Yes, to achieve harmonized, quality-assured data
granularity single records, granular
content Those fields/ InfoObjects, which can be harmonized and/ or quality assured
history Often no persistency – Lookup-Mappings: long term
main services Provide consistent, harmonized data across multiple sourcesProvide quality assured data
store & deploy often only virtual as InfoSource or just as Transformation RuleIf explicit persitency needed ‚normal‘ DSO with overwriteLookup tables as DSOs/ InfoObjects/ Z-tables (history!)
update Request by request
housekeeping
Impl
emen
t.O
p.D
WH
Ser
vice
s
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /20090210/ Page 91
Detailing LSA Reference LayersPropagator Layer I
SourceSystems
EDW Layers
Data flow
Corporate Memory
BusinessTransformation
Layer
ReportingLayer
AcquisitionLayer
Data Mart LayersHarmonize/
QualityV-
Layer
DM1
DM3
DM2
Propagator LayerThe Propagator Layer is the interface to all business related Layers. It stands for ‘extract once deploy
many’ i.e. scalability of Data Mart developmentPropagator Layer Objects offer data that are easy to digest, ready to consume for all business
purposes (Data Marts internal & external)
PropagationLayer
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 92
PropagationLayer
Detailing LSA Reference LayersPropagator Layer II
SourceSystems
EDW Layers
Data flow
Corporate Memory
BusinessTransformation
Layer
ReportingLayer
AcquisitionLayer
Data Mart LayersHarmonize/
QualityV-
Layer
DM1
DM3
DM2
Easy to digest means standardized, unified data :From Harmonization & Quality Layer transformations we get:
Compliance of data to corporate quality and consistency standardsIntegration of local master data to group master data for group reportingIntegration of local master data into a single BW data model (compounding)
Easy to digest means standardized, unified data :Propagator Layer adds:
Option to enhance or merge Data to simplify Data Mart building - but no business specifictransformations (no flavor for all EDW Layers)
Unified Data transfer behavior out of Propagator Objects (e.g. additive delta)Suitable portions of data (-> Domains) to fulfill SLAs related to local autonomy, report
availability, robustness, recovery, administration and operations
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 93
Detailing LSA Reference LayersPropagation Layer & Trimming Data
DataSource ASource 2
DataSource ASource 1
‘DataSource A’Propagator
3. Collect
DataSource BDataSource A
‘DataSource A & B’Propagator
2.Merge
DataSource A
‘DataSource A +’Propagator
1.ExtendAdddata
DataSource A
‘DataSource A’Propagator 1
4.Split
‘DataSource A’Propagator 2
Propagator DSOs: Unified BI application experience – additive delta
viaqueue
viageneric
viafull
moving
via full
completefull
incompletefull ?
5.Unify
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 94
Detailing LSA Reference LayerPropagation Layer & Digestible Data
The core service of a Propagation Layer is to offer digestible data to applications.
Digestible may mean
harmonized data in the broadest sense (for more details s. Chapter on Data Model)integrating data: common semantics, common valuessmoothing data: common semantics, technically unified values (e.g. compounding)
harmonize behavior of extractors (additive delta to Data Marts)
trimmed to fit DataSources and Data persistency‘s toReduce data complexity for applications
1. Extending DataSources by looking up information, which applications frequently askfor. Note: introduced parent-child relationship must be stable otherwise realignmentissues!
2. Merging different but highly related DataSources and store data in a single propagator,If application always or frequently request them together (e.g. HR InfoTypes, avoidingextractor enhancements)
Provide sound data portions for better support of application services (availability etc)3. Collecting data from the same (or similar) DataSource but from different source
systems to less or a single source system independent propagator (s) ( LSAdomains)
4. Splitting data from a DataSources into multiple persistency’s with identical structure( LSA domains)
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 95
Data flow
2LIS_11_VASTI
2LIS_11_VASCL
YGTCS_SUMMARY
2LIS_12_VCSCL
Acquisition
Corp Memory
EDWpropagationR/3
Corp Memory
Corp Memory
Corp Memory
EDW LayerCustomer Example
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 96
LSA Reference LayerData Propagation Layer Flier
Criteria Characteristics
potential sources Data Acquisition Layer, Harmonization Layer, Corporate Memory
potential targets Business Transformation Layer, Reporting Layer
reusability Yes
transformations Additional, stable fields to increase (re-)useability & accessibility.No application-specific rules!
granularity single records, granularity defined by DataSource business-keycontent DataSource specific
as comprehensive as possible, if propagator is expecting volatile requiermentsMerge of different DataSources to reduce complexity changes
history Minimum history defined by requirements of target-applications/ dependent fromCorporate Memory existence
main services ‘Single Point of Truth’ for BI applications (Business Transf. & Reporting Layer)Provide digestible (additive delta, content, performance) data for BI applicationsapplication recovery, rebuilt
store & deploy ‚normal‘ DSO in overwritesemantical/ logical partitioned for large scale DWH/ time-zone support
update driven by BI application requirements (report availability)
housekeeping Regular delete of DSO change-log content
DW
H S
ervi
ces
Impl
.O
p.
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /20090210/ Page 97
LSA Reference LayersData Mart Layers
SourceSystems
EDW Layers
Data flow
Corporate Memory
BusinessTransformation
Layer
ReportingLayer
PropagationLayer
AcquisitionLayer
Data Mart LayersHarmonize/
QualityV-
Layer
DM1
DM3
DM2
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 98
Data flow
2LIS_11_VASTI
2LIS_11_VASCL
YGTCS_SUMMARY
2LIS_12_VCSCL
Acquisition
Corp Memory
InfoSource
InfoSource
ReportingEDW
propagationR/3Appl specific
Transformation
Corp Memory
Corp Memory
Corp Memory
Acquisition/Pass-Thru
Corporate Mem.
Propagation
Transformation
Reporting
Multi-provider
Corporate Mem.
Business Transformation LayerCustomer Example
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 99
LSA Reference LayerBusiness Transformation Layer Flier
Criteria Characteristics
potential sources Data Propagation Layer
potential targets Reporting Layer
reusability No
transformations Applications specific rules, aggregation, disaggregation, lookups
granularity Defined by application needscontent Defined by application needshistory Defined by application needs
main services area where complex merging of propagators objects will take placearea where complex calculations/ enrichment (lookups) will take placeguarantees reporting layer consistency (e.g. Synchronization of DataSources)
store & deploy ‚normal‘ DSO in overwrite, if any: often only virtual as InfoSource/Transf. Rulesemantical/ logical partitioned for large scale DWH/ time-zone support
update driven by BI application requirements (report availability)
housekeeping Regular delete of DSO change-log content
DW
H S
ervi
ces
Impl
.O
p.
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /20090210/ Page 100
Detailing LSA Reference LayersSub-Layers of Reporting/ Data Mart Layer
Access Abstraction-/Virtual Layer
Reporting/ Data Mart Layer
Analytics-/Dimensional Layer
Flexible Reporting/Granular Reporting
Layer
highly granularhighly comprehensiveshort life cycleflat, multidimensional
less granularless comprehensivelong life cyclemultidimensionaloptimized performance
abstract from physics‘virtual’flexible ‘view’ generationprotect front-end investments
Planning Layer
dedicated for planningdirect input structures
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 101
Detailing the Reporting LayerCustomer Example
FlexibleLayer
DimensionalLayer
VirtualLayer
ReportingGranular
ReportingPerformance
Long term
AbstractionFlexible
Data flow
Reporting Layer
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 102
© SAP 2008 / Page 103
LSA Reference LayerReporting Layer Flier
Criteria Characteristics
potential sources Data Propagation Layer, Business Transformation Layer
potential targets -
reusability No
transformations Applications specific rules, aggregation, disaggregation, lookups
granularity Defined by application needscontent Defined by application needshistory Defined by application needs
main services reporting & analysissave investments in front-end area: separation of physical Objects (InfoCube,
DSO) and virtual Objects( Virtual InfoProvider, InfoSet) by using logical views(MultiProvider)
different services of Reporting Layer often modeled by sub-LayerAccess Abstraction Layer (MultiProvider logical view)Analytic Layer (InfoCubes)Flexible Reporting (granular InfoCubes, DSOs)
store & deploy InfoCubes, DSO-Objects, InfoSets, Virtual InfoProvider, MultiProviderQueries work always on MutliProvidersemantical/ logical partitioned for large scale DWH/ time-zone support
update driven by BI application requirements (report availability)
housekeeping Archiving, NLS, Compression
Impl
.O
p.D
WH
Ser
vice
s
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /20090210/ Page 103
LSA Reference Layer –Operational Data Store (ODS)
LSA
Reporting Layer (Architected Data Marts)
Business Transformation Layer
Operational D
ata Store
Data Propagation Layer
Quality & Harmonisation Layer
CorporateMemory
Data Acquisition Layer
Virtualization Layer
EDW layerSingle Point of truthReusableCorporate ownedGranular
ArchitectedData Mart Layer
User
Sources
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 104
3rd PartyLandscape
SAP
BW
PIPEPOS Inbound Processing Engine
Upload of POS Data to SAP BWvia POS Inbound Processing Engine
mySAPLandscape
Sales Analyses of POS Data
StoreContr.
PromoAnalysis
LossPrevent
BI-R
epor
ting
Tem
plat
es
Outbound Interfaces
Retail High Volume SAP BW ODS – POS DataManagement
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 105
3rd PartyLandscape
SAP BW
mySAPLandscape
Reporting-Templates
SAP BWMasterdata
ExchangeInfrastructure
R/3 POS Inbound
Card Payment
Billing
Forecast &Replenisment
InventoryManagement
SAP BW
Custom-BuiltInterface
IDocs
StandardsSales
TransactionsVMIXML
Transactiondatabase
Sales Audit• Monitor• Analyses• Editor
BAPI
PIPE Access Module
Overview POS Inbound Processing Engine(PIPE)
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 106
Example for Dedicated SAP BW OperationalData Stores
SAP BW
PIPE-ODS
APA
SAP BW
PIPE-ODS
Americas
SAP BW
PIPE-ODS
UK
SAP BW
PIPE-ODS
EMEA
CentralSAP BW
CentralSAP for Retail
Master DataPOS
TransactionData
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 107
Hybrid ProviderReal-Time Business Intelligence
HybridProviderCombines mass data with latest deltainformation at query runtimeConsists of a DataStore object and aInfoCube with automatically generateddata flow in betweenDSO object can be connected to a realtimedata acquisition DataSource/DTPIf the DataSource can provide appropriatedelta information in direct access mode aVirtualProvider can be used instead of theDSO.Facilitates replication of DSO-/VirtualProvider data to SAP NetWeaverBW Accelerator by switching off databasepersistency of the InfoCube
(RDA) DTP
ReplicateDSO
latest data mass data
InfoCube
HybridProvider
OLTP
Implement operational reporting scenarios leveraging newmodeling objects
BWA
optionally
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 108
LSA Reference LayerOperational Data Store Layer Flier
Criteria Characteristics
potential sources Data Acquisition Layer
potential targets application specific (Reporting Layer), external (e.g. Pipe)
reusability Limited
transformations simple
granularity extracted records granularitycontent application specific
history application specific
main services near real time reportingmass data management (e.g. POS)
store & deploy normal DSO (Hybrid Provider)mass data
Write-optimized (wo) DSONo PSA
Pipe (point of sale inbound processing engine)update RDA, request by request
DW
H S
ervi
ces
Impl
emen
tO
p.
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /20090210/ Page 109
LSA Reference LayerAcquisition Layer and Subsequent Layer
Why are there different paths thru LSA? The general answer is simple:In a BW we have to fulfill different competing SLAs (Service Level Agreements)Experience shows that a single staging approach with single persistent stores for allpurposes cannot achieve this (RDBMS limits) ! This applies the more challenging theconditions are for BW.Note: This means on the other hand side that if we found less complex conditions the LSABuilding Blocks set up can be simpler. This applies as well for an overall customer LSA set upas for specific data sources within a full blown customer LSA !
What are complex conditions andchallenging requirements?
24x7, time zoneshigh volume, not split able loadssmall recovery window (e.g. 6h)out sourcing (skills?)off shoring (skills?)high operational robustnesshigh report availability (e.g. >96%)high application flexibility....
LSA
Reporting Layer (Architected Data Marts) –Shared Master Data (InfoObjects)
Business Transformation Layer
Operational D
ata Store
Data Propagation Layer
Harmonization LayerCM
Data Acquisition Layer
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 110
LSA Layers @CustomerSummary
The LSA describes services that are relevant at large scale BWimplementations. The services are grouped by Layers.
The LSA suggested Layers are a basic offering. Based on customersituation and preferences additional ones may be introduced evenlater on (Customer LSA).
As with all LSA areas customer adaptation of Layers follows the80:20 rule i.e. concentrate on services first, which offer the mostbenefit for the majority of data
Not all Layers have necessarily own persistent InfoProviders for alldata flows. Whether and when persistent InfoProviders exist are givenin the LSA Implementation standards
The Customer LSA Layer definitions apply throughout the BW EDW– No exceptions without approval!
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 111
BI-Projekt-Design
The Layered Scalable ArchitectureCustomer Adaptation & Utilization
SAP LSA: The Reference Architecture
Customer LSA : Standards - Handbook
BI-Projekt-DesignCustomer BI Projects (Plan-Build-Run)
Step 3:PerfectPerfect
Customer LSA
Step 4:UpdateUpdate
Customer LSA
Step 1:DesignDesign
Customer LSA
Step 2:ApplyApply
Customer LSA
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /20090210/ Page 112
LSA Building BlocksReference LSA & Customer LSA – Example I
LSA
Reporting Layer (ArchitectedData Marts)
Business Transformation Layer
Operational D
ata Store
Data PropagationLayer
Quality &Harmonisation Layer
Corporate
Memory
Data Acquisition Layer
From Reference LSA to Customer LSA:Not all LayerAdditional LayerDifferent visualization and branding of Layer
Reference LSA Customer LSA
SAP
Sour
ces
Oth
erSo
urce
s
Operational Data Store
Dat
a A
cqui
sitio
n
Dat
a Pr
opag
atio
n
Qua
lity
Tran
sfor
mat
ions
Bus
ines
s Tr
ansf
orm
atio
n
Flex
ible
Rep
ortin
g
SubsidiaryReporting &
Analytics
Business-UnitSpanning
Reporting &Analytics
ScorecardsGroup
Near real timeReporting
Acc
ess
Abs
trac
tion
Laye
r
Proj
ect-d
efin
ed R
epor
ting
& A
naly
tics
End-
user
Rep
ortin
g &
Ana
lytic
s
CorporateMemory
EnterpriseData Warehouse BI Application Area
Data flow
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 113
LSA Building BlocksReference LSA & Customer LSA – Example I
LSA
Reporting Layer (ArchitectedData Marts)
Business Transformation Layer
Operational D
ata Store
Data PropagationLayer
Quality &Harmonisation Layer
Corporate
Memory
Data Acquisition Layer
From Reference LSA to Customer LSA:Detailing
Reference LSA Customer LSA
AcquisitionLayer
CorporateMemoryLayer
PropagationLayer
BusinessTransformation
Layer
FlexibleLayer
DimensionalLayer
VirtualLayer
(YADSSnnn)
YCDSSnnn
YPDSSnnn YBAPPnnn
YFAPPnnn
YVAPP1nn
YVAPP2nn
YDAPPnnn
D a t a f l o wlookup
1:1Unlink
UnflavoredIntegratedGranular
Ready to consume
Applybusiness-logic
Reporting
Granular
ReportingPerformance
Long term
AbstractionFlexible
CompleteComprehensiveMost granular
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 114
LSA Building BlocksReference LSA & Customer LSA – Example II
LSA
Reporting Layer (Architected DataMarts)
Business Transformation Layer
Operational D
ata Store
Data Propagation Layer
Quality & HarmonisationLayer
Corporate
Memory
Data Acquisition Layer
From Reference LSA to Customer LSA:DetailingNot all LayerAdditional LayerDifferent visualization and branding of LayerIndividual domains
Reference LSA Customer LSA
Data flow
2LIS_11_VASTI
2LIS_11_VASCL
YGTCS_SUMMARY
2LIS_12_VCSCL
Acquisition
Corp Memory
ReportingEDWpropagationR/3
Appl specificTransformation
Corp Memory
Corp Memory
Corp MemoryCorporate Mem.
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 115
The Layered, Scalable Architecture (LSA) for EDW
Reference LSA & Customer LSA3.I3.I
33
Agenda PDEBW1
LSA Building Blocks – Data Layer3.III3.III
LSA Building Blocks – Data Domains3.V3.V
LSA Building Blocks – Data Model3.VI3.VI
LSA Building Blocks – Layer & Master Data3.IV3.IV
Reference LSA Overview3.II3.II
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 116
Situation of Master Data
With master data we have similar challenges as with transaction data and additional ones
LSA topics1. High volume master data tables have a load impact
1. With full loads for InfoObjects (even if delta-load possible), which are used with extractorenhancements to catch the changes of an enhancement ( merging DataSources inPropagator Layer)
2. With change run (aggregate maintenance)2. Scenario reporting readiness depends on master data load time
1. Transaction data can only be processed into target subsequent to master data loads2. Different scenario or market reporting readiness requires different points in time of master
data readiness3. Recovery of InfoCubes or new ones often need master data history
BIA topics1. Change Run2. High volume master data tables (shared dimensions) have a negative impact on query
performance as with large fact tables3. Limitations of external hierarchies on high volume master data
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 117
LSA & Master DataSituation
Master data have two main tasks to fulfill:– Being target of lookups during transactional data load (referential integrity, adding
information– Being the shared dimensions of InfoCubes for reporting (MultiProvider
Today InfoObject hosted master data (P,Q,S,X,Y tables) serve for both purposes.
InfoObjectMaster data
Shared Dimension(Navigational Attributes)
Integrity,Lookups
Transactional loads
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 118
Issues with Data Mart Level Managed Master Data
Process C
hainD
ata MartA
Transactionalloads
With non-architected BWs the master data loadsare often managed on BI application/ Data Martlevel what leads to redundant, uncontrolledmaster data processing.This is unacceptable for large scale BWs
Data Mart level managed master data:
Master dataLoad e.g.
0CUST_SALES+ Change Run
3:00
Process ChainMaster Data
Master dataLoad e.g.
0CUST_SALES+ Change Run
Transactionalloads
First Step: BW level managed master data:
Master data should be collected/ prioritized andloaded across the Data Marts thus become BWmanaged
Transactionalloads
Master dataLoad e.g.
0CUST_SALES+ Change Run
Process C
hainD
ata MartB
1:00 1:15
Transactionalloads
Process ChainData Mart B
Process ChainData Mart A
2:301:00 1:15
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 119
LSA ImplementationLayering Master Data
LSA
Reporting Layer(Architected Data Marts)
Propagation/CM Layer
Quality & Harmonisation Layer
Data Acquisition Layer
InfoObject tables
Master DataDSO
Shared Dimensionx,y,p,q,s tables
(Navigational Attributes)
Integrity,Lookups
Picture shows master data with delta load, master data withfull loads need additional ‘assistant DSO’ to determine delta
Large scale BWs should layerthe master data:
1.Separation of staging andreporting tasks
InfoObject tables forreportingPropagator Master DataDSOs for staging
2.Storing master data history(introducing ‘active from’‘active-to’ time-slice inPropagator Master DataDSO)
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 120
LSA ImplementationDecoupling EDW & Data Mart Layer Loads
Process ChainsMaster Data
InfoObjectChange RunPropagator/
Corporate MemProcess ChainData Mart A
LSA managed master data loads
EDW transactional & master data loads are decoupled from Data Mart &InfoObject loadsIt still remains challenging at what point in time to schedule master dataloads in a global system
Process ChainsEDW Master Data
Propagator for0CUST_SALES
Process ChainsEDW Transactional
Data
Data Mart LayerB Trans Layer
InfoObjectUpdate
EDW Layers Data Mart Layers
Process ChainData Mart B
Data Mart LayerB Trans Layer
multiple loads
per day
loads by
business
requirements
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 121
LSA ImplementationReal Time Data Acquisition (RDA) for Master Data
Process ChainsMaster Data
InfoObjectChange Run
Propagator/Corporate Mem
Process ChainData Mart A
LSA managed master data loads (RDA)
With NW BW 7.30 there will be Real Time Data Acquisition (RDA) forMaster Data .This will increase the robustness of global BW EDWs considerably
RDAEDW Master Data
Propagator for0CUST_SALES
Process ChainsEDW Transactional Data
Data Mart LayerB Trans Layer
InfoObjectUpdate
EDW Layers Data Mart Layers
Process ChainData Mart B
Data Mart LayerB Trans Layer
multiple loads
per day loads by
business
requirements
continuous loads
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 122
LSA and Master DataKeep master data history
Customer CustomerGroup
A Y after 2nd Load
Customer CustomerGroup
A X 1st Load: 20020101A Y 2nd Load: 20020401
InfoObject Master Data Table
Master Data from Source or Master Data Server
InfoObject master data keep only the actual version of master data:
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 123
LSA and Complete History of Master Data
Corporate Memory DS-Objects store all versions of master data
customer mother-compcustomer mother-comp
AAA YBBB YCCC X
InfoObject Customer Master
customer ActiveFrom mother-compcustomer ActiveFrom mother-comp
AAA 19980101 XAAA 19981101 YBBB 19980101 YCCC 19980101 X
I.Simple Customer Master CM
DS-Object
customer mother-comp
AAA XBBB YCCC X
customer mother-comp
AAA Y
load from 19980101
load from 19981101
customer ActiveTo ActiveFrom mother-compcustomer ActiveTo ActiveFrom mother-comp
AAA 19981031 19980101 XAAA 99991231 19981101 YBBB 99991231 19980101 YCCC 99991231 19980101 X
II.Time slice Customer Master CM
DS-Object
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 124
Alternatives Managing Master Data in LSA
1. Slim Staging: direct load into InfoObject (like in normal BW) for all Master Data notaddressed in 2 or 3.
2. Complete history of master data changes in a Propagator/ Corporate Memory* viatime-stamping records – (restricted usability)
Flexible stagingDelta loads direct into CM DSOFull loads via intermediate normal DSO (special type of Pass Thru) for delta determination
Same handling for transactional full loads
3. Complete history of master data changes in a Propagator/ Corporate Memory* viatime-sliced history (easy accessable in loockup)
Flexible stagingPropagator/ CM normal DSO with active-from, active-to (via function module set) slice, active-to is part of key
Delta loads direct into Propagator/ CM DSOFull loads via intermediate normal DSO (special type of Pass Thru) for delta determination
* whether you make it a Propagator or a CM member is a matter of taste© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 125
SAP BW EDW Functions and ModelingDetermine Time slice of Master Data I
InfoObject
Characteristic Active-to Active-from Origin Attribute1 Attr24711 99991231 20030501 S1 Y A4711 20030430 20030401 S1 X A4712 99991231 20030401 S1 X B
Characteristic Origin Attribute1 Attr24711 S1 X->Y A4712 S1 X B
Characteristic Attribute1 Attr24711 Y A4712 X B
Transformation Rule:• Read DSO-Object with 4711 and 99991231• Update 99991231 record with new values and Active-fr• Use Return table to create new record
Characteristic Attribute1 Attr24711 X->Y A4712 X B
Corporate Memory‘normal DSO’
‘Pass Thru’‘normal DSO’
Load (here full)
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 126
LSA and Master Data
Completeness of Master Data History
At least for the ‘big entities’ all historical master data versionsshould be stored in the SAP BW Data Warehouse Layer usingflexible master data staging.For less important entities it is sufficient to keep the actualversions of master data in the InfoObject master data table thususing direct staging.
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 127
The Layered, Scalable Architecture (LSA) for EDW
Reference LSA & Customer LSA3.I3.I
33
Agenda PDEBW1
LSA Building Blocks – Data Layer3.III3.III
LSA Building Blocks – Data Domains3.V3.V
LSA Building Blocks – Data Model3.VI3.VI
LSA Building Blocks – Layer & Master Data3.IV3.IV
Reference LSA Overview3.II3.II
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 128
Incremental EDWReasons & Challenges
Incremental set up is targeting to:
Align actual business requirements & BI/ EDW excellence
Reduce complexity
Step by step process & organizational excellence
...
Nevertheless an incremental approach is challenging as we have to guaranteeconsistency & scalability during the stony path of roll-out:
Consistency– Between the various Data Marts (FI, CO, LO, CRM...)
– consistent development of Data Marts– Sometimes between different SAP BW systems
– consistent deployment of Data MartsScalability
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 129
EDW Life Cycle DimensionsIncremental Implementation & Scalability
2. Incremental in terms of organizational coverage – organizational roll-out
Dem
and Planning
Generating D
emand
Procure 2 P
ay
CO
PA
..... .....
EDW
nBI Application Coverage & EDW
Org U
nit A
Org U
nit B
Org U
nit C
Org U
nit N
..... .....
EDW
nOrganizational Coverage & EDW
Needs Scalabilitythat is addressed by
(EDW)- layers
Needs Scalabilitythat is addressed by
domains (& datamodel)
1. Incremental in terms of functional coverage – BI application/ Data Mart roll-outAn EDW implementation is always an incremental one, never a big-bang
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 130
LSA Reference Layers Highly Simplified -Layering Does Not Resolve All Challenges
SourceSystems
EDW Layers
Data flow
Corporate Memory
BusinessTransformation
Layer
ReportingLayer
PropagationLayer
AcquisitionLayer
Data Mart LayersHarmonize/
QualityV-
Layer
How to support:a world wide continuous roll-outBW-wide load scalability24x7 operations & administrationdifferent reliability of sourceslocal autonomyoverall local & group SLAs...
?© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 131
LSA Landmark Building BlocksData Domains
Non-Architectured
D a
t a
f l o
w
D a
t a
f l o
w
LSAArchitectured
Layer
Domain
Domains means structuring/ modeling of data within the layers:Transparent, disjunct structuring of transactional data using stable criterion.Target is the support of:
Independency/ autonomy of organizations24x7, time-zonesScalability / performance/ low latency(parallel vers sequential)Challenging recovery-windowEmbedding into RDBMSImplementation & Operational robustness
Value of Data Domains:+ Transparency & Flexibility
+Development, Maintenance+Administration, Operations
+ Scalability & Robustness+Application+Load/ Query Performance+Database-Integration
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 132
LSA Domain Concept - Strategic BW PartitioningAccross the Layers – For All Flows I
SourceSystems
EDW Layers
Data flow
Corporate Memory
BusinessTransformation
Layer
ReportingLayer
PropagationLayer
AcquisitionLayer
Data Mart LayersHarmonize/
QualityV-
Layer
DM1
Domains apply to transactional dataNormally not to master data!
2LIS_11_VAITM
Belongs to Domain A
Belongs to Domain B
Belongs to Domain C
Single ER
P
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 133
LSA Domain Concept - Strategic BW PartitioningAccross the Layers – For All Flows II
SourceSystems
EDW Layers
Data flow
Corporate Memory
BusinessTransformation
Layer
ReportingLayer
PropagationLayer
AcquisitionLayer
Data Mart LayersHarmonize/
QualityV-
Layer
DM1
Domains apply to transactional dataNormally not to master data!
Belongs to Domain A
Belongs to Domain B
Belongs to Domain C
2LIS_11_VAITM
Multiple E
RP
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 134
Domain US
LSA Building Blocks &Extract Once - Deploy Many
EDW
Propagators
Data Marts
DataSources
Ord
er
Del
iver
y
Sup
ply
Cha
in
Ord
er In
fo
1:n
Del
iver
y In
fo
Domain AMS
Ord
er
Del
iver
y
Sup
ply
Cha
in
Ord
er In
fo
Del
iver
y In
fo
Domain APJ
Ord
er
Del
iver
y
Sup
ply
Cha
in
Ord
er In
fo
Del
iver
y In
fo
Filling Domains with respect to domain criteria
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 135
LSA Data DomainsBW Landscape Consolidation (Consolidation BW)
EuropeJapan
Asia Pacific
ERPERPERPERPERPERP
ERPERP
ERPERP
BW BW
BW
BW
BW ERPERP
North America
South America
A BW EDW replaces a bunch of existing BWs and/ or legacy DWHs(BI Consolidation) spread across the organization
To enable comparable services like we had in a distributed, multiple DWH instance world(yes, there are some nice things) we introduce Data Domains in a BW EDW that divide
the transactional data but use identical meta data & master data.
BW
Using Domains in a BW EDW stands for manageability & flexibilityDomains allow SLAs in a BW EDW like in a distributed BW world
X
X
Domain AmericasDomain Americas
XX
Domain EuropeDomain Europe
X
X
Domain Asia PacificDomain Asia PacificBW EDWBW EDW
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 136
LSA Data DomainsBW in Parallel to ERP Rollout (Primary BW)
EuropeJapan
Asia Pacific
North America
South America
A single BW EDW shall offer standard reporting & analytics for all organizational unitsin a global ERP rollout.
To enable comparable services like we had in a distributed, multiple BW instanceworld we introduce Data Domains in a BW EDW that
divide the transactional data but use identical meta data & master data.
Using Domains in a BW EDW stands for manageability & flexibilityDomains allow SLAs in a BW EDW like in a distributed BW world
AMSAMS GermanyGermany APAAPAUSUS EMEAEMEA
Global ERPGlobal ERP
BW EDWBW EDW
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 137
LSA Data DomainsDivide Data by Sources-‘Quality’ (Integration BW)
BW EDWBW EDW
mainERP
remoteERP 1
remoteERP 2
Main DomainMain Domain Remote DomainRemote Domain
less stable, no controlstable, controlled
Using Domains in a BW EDW stands for manageability & flexibilityDomains allow SLAs in a BW EDW like in a distributed BW world
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 138
1st Summary: Domains Targets to Keep Large BWEDWs Manageable & Scalable
Dom
ain
Wes
tD
omai
n W
est
Dom
ain
Cen
tral
Dom
ain
Cen
tral
Dom
ain
East
Dom
ain
East
BW EDWBW EDWCompany
is operatingIn North America
Using Domains in a BW EDW stands for manageability & flexibilityDomains allow SLAs in a BW EDW like in a distributed BW world
North America
South America
Domain SouthDomain SouthAMSAMS
Company isexpanding to
South America
Dom
ain
Dom
ain
Wes
tW
est
Dom
ain
Dom
ain
Cen
tral
Cen
tral
Dom
ain
Dom
ain
East
East
BW EDWBW EDW
EuropeNorth America
South America
Domain SouthDomain SouthAMSAMS
Dom
ain
Dom
ain
Wes
tW
est
Dom
ain
Dom
ain
Cen
tral
Cen
tral
Dom
ain
Dom
ain
East
East
DomainDomainEUEU
Company isexpanding to
Europe
BW EDWBW EDW
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 139
LSA Data Domains BackgroundEvolutionary BW EDW
An BW EDW that has to provide ‘local’ BI services facesby far more databy far more and more complete BI applications – finally the entire value-chainmore source systemshigher query workload
than any other DWH in the organization before!
But with service expectations necessary to support daily business, like:High performance, robustness, high availability, low latency, organizational autonomy, fastrecovery ......
In addition BW EDWs on a global scale have to observe time-zones and24x7 operations
These challenges cannot be answered by just layering data: The LSA answeris the Data Domain concept:
LSA Data Domains divide (transactional) data in the BW Layer in a disjunctiveas stable as possible manner enabling scalable EDW services for most BI
needs across the business© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 140
LSA Building BlocksReference LSA & Customer LSA – Example
Acquisition/PSA Layer
CorporateMemoryLayer
PropagationLayer
BusinessTransformation
Layer
FlexibleLayer
DimensionalLayer
VirtualLayer
(YADSS100)
YCDSS100
YPDSS1DX
YPDSS1GX
YPDSS1WX
YPDSS1UX
YBAPP1AX
YBAPP1DX
YBAPP1GX
YBAPP1WX
YBAPP1UX
YFAPP1AX
YFAPP1DX
YFAPP1GX
YFAPP1WX
YFAPP1UX
YDAPP1AX
YDAPP1UX
YVAPP1XX
YVAPP1XXYDAPP1WX
YDAPP1DX
YDAPP1GX
Lookup-tables
Asia
Europe
Americas
Germany
US
YPDSS1AX
Controltables
LSA
Reporting Layer (Architected DataMarts)
Business Transformation Layer
Operational D
ata Store
Data Propagation Layer
Quality & HarmonisationLayer
Corporate
Memory
Data Acquisition Layer
From Reference LSA to Customer LSA:Individual Domains
Reference LSA Customer LSA
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 141
Transparent, Scalable Structuring of BW:LSA Domains & Layers
LSA
Reporting Layer (Architected Data Marts)
Business Transformation Layer
Data Propagation Layer
Quality & Harmonisation Layer
Corporate
Memory
Data Acquisition Layer
Access Abstraction Layer(MultiProvider)
Single Source System (Layer)
LSA
Reporting Layer (Architected Data Marts)
Business Transformation Layer
Data Propagation Layer
Quality & Harmonisation Layer
Corporate
Memory
Data Acquisition Layer
Access Abstraction Layer(MultiProvider)
Multiple Source Systems (Layer)
Distributionactively designed:
Domains
Distributioninherited
LSA Domains distribute the transactional data for the entire BW EDW in adisjunctive manner. The meta data of domains are identical.
The LSA addresses an evolutionary EDW approach introducing DataDomains enabling ‘local’ BI services, 24x7 operations andadministration without neglecting the broader EDW picture.
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /20090210/ Page 142
LSA Data Domains and LayerProperties of LSA Domains
As a rule of thumb:
Domains organize data by a general criteria driven by Reporting, BI andManageability requirements i.e. domains often differ from operational dataorganizationDomains are from transactional data point of view disjunct – harmonized masterdata are sharedDomains use identical meta data definitionsCross views are achieved via MultiProviders or dedicated persistent InfoProvidersDomains should be stableDomains should be consistent across all Layer (across all flows)Domains should be general for all Layer, exceptions:
The Acquisition Layer inherits the structuring from source systems/ extractorsDomains do not apply on the Corporate Memory
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 143
Criteria Defining LSA Domains
Golden rule defining domains:
as many domains as necessary – as less domains as possible
Defining Domains – rules of thumb
Often a geography driven domain concept works wellOften one basic domain per continent makes sense ( rough time zone handling)e.g. APA, EMEA, Americastime zone aspects may lead to e.g. 3 Asian domains East-Asia, Mid-Asia, West-AsiaE.g. for continental BW EDW a starting point could be East, Middle, West...
Expected Volume (realistic volume estimates are a key input defining domains )Large APA volume contribution & large (for your business important) countries may get anown domain e.g. China and US
Independency (special service level agreement) for certain countriesImportant countries/ markets get an own domain
Robustness: take Quality/ stability of different sources into consideration(potential) instable data offerings from sources may lead to an own domain
Each customer may have additional criteria, normally it is a mixture of multiple items
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 144
LSA Domain Criteria & Business View
Domains should be as stable as possible (TCO) on the one hand side but onthe other hand side from the overall service perspective it makes sensedefining domains with respect the business view on data, but this is notnecessarily stable as it refers to organization.
We may find a business view of data that is not reflected by organizationalunits of the sources (e.g. sources know Company-Code, planning area etc butbusiness view is ‘market’ what is not reflected by the sources)
Domains offer the opportunity resolving this using domains in an EDW forbetter BI services (reporting) and more transparency (domains)
Note: As domains are a mixture of application & technical dimensions weshould not overload domains with a purely business view -> observe Goldenrule
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 145
Domains and Source SystemsDefining Adequate Domains
...... ......
......
......Acquisition
Propagation
Acquisition
Propagation
DataSource from single corporate source-system
DataSource from multiple source-systems
?
??
Nodomains
Nodomains
adequatedomains
adequatedomains
Too manydomains
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 146
LSA Domains Definitions & EDW Roll-OutManaging Volume Uncertainty
Especially at the beginning of an EDW roll-out there is often a large degree ofuncertainty with respects to volume expected in the future.
But there is normally at least a classification of organizational units that drive thedomains possible in terms of t-shirt sizes (S,M,L)
S & M organizational units reside in standard domains (ASIA; AMERICAS; EMEA)
L organizations/ markets get an own domain
Possible challenges during roll-out:
Domains will become too big e.g. EMEA -> Create an additional Domain EMEA2
Large Domains -> think about additional splitting criteria (note: time-partitioning helpsnormally little with load issues)
This leads to initial domain concept for your EDWe.g. Asia, EMEA, AMERICAS, CHINA, US
Note: Existing Domains must continuously be observed with respect to SLA compliance
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 147
LSA DomainsDomain DWH Characteristic
BW EDWAll data
Domain ‘B’EMEA
Domain ‘A’APA
Domain ‘C’AMS
Country/ MarketE.g. ‘F’
E.g. ‘UK’
,,,,,,,,
Organizational-unit0COMPCODE = FR01
0COMPCODE = FR02
In BW EDW all loadedrecords will be qualified/‘stamped’ assigning a stable‘domain- driving’characteristic like market/country to organizationalcriteria like in this example0COMPCODE coming fromsource system.
This allows easy redistribution of adomain data if service levelscannot be kept
assign
assign
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 148
LSA DomainsDomain DWH Characteristic – New Domain
BW EDWAll data
Domain ‘B’EMEA – ‘F’
Domain ‘A’APA
Domain ‘C’(AMS)
E.g. ‘UK’
,,,,,,,,
Organizational-unit0COMPCODE = FR01
0COMPCODE = FR02
Domain ‘F’Country ‘F’
France ‘F’ gets an owndomainAll records in Domain ‘B’with country ‘F’ are movedto the new domain
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 149
LSA & Importance of Volume Estimates
© SAP 2009 / PDEBW1- EDW & BW Layered, Scalable Architecture (LSA), Juergen Haupt /200910/ Page 150
top related