1 1 the it perspective: data warehousing, management, and analytical structures rafal lukawiecki...

30
1 1 The IT Perspective: Data Warehousing, Management, and Analytical Structures Rafal Lukawiecki Strategic Consultant, Project Botticelli Ltd [email protected]

Upload: kory-perkins

Post on 23-Dec-2015

228 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: 1 1 The IT Perspective: Data Warehousing, Management, and Analytical Structures Rafal Lukawiecki Strategic Consultant, Project Botticelli Ltd rafal@projectbotticelli.com

11

The IT Perspective: Data Warehousing, Management, and Analytical Structures

Rafal LukawieckiStrategic Consultant, Project Botticelli [email protected]

Page 2: 1 1 The IT Perspective: Data Warehousing, Management, and Analytical Structures Rafal Lukawiecki Strategic Consultant, Project Botticelli Ltd rafal@projectbotticelli.com

22

ObjectivesExplain the basics of:1. Master Data Management2. Data Warehousing3. ETL4. OLAP/Multidimensional Data

The information herein is for informational purposes only and represents the opinions and views of Project Botticelli and/or Rafal Lukawiecki. The material presented is not certain and may vary based on several factors. Microsoft makes no warranties, express, implied or statutory, as to the information in this presentation.

Portions © 2010 Project Botticelli Ltd & entire material © 2010 Microsoft Corp. Some slides contain quotations from copyrighted materials by other authors, as individually attributed or as already covered by Microsoft Copyright ownerships. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Project Botticelli Ltd as of the date of this presentation. Because Project Botticelli & Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft and Project Botticelli cannot guarantee the accuracy of any information provided after the date of this presentation. Project Botticelli makes no warranties, express, implied or statutory, as to the information in this presentation. E&OE.

This seminar is based on a number of sources including a few dozen of Microsoft-owned presentations, used with permission. Thank you to Chris Dial, Tara Seppa, Aydin Gencler, Ivan Kosyakov, Bryan Bredehoeft, Marin Bezic, and Donald Farmer with his entire team for all the support.

Page 3: 1 1 The IT Perspective: Data Warehousing, Management, and Analytical Structures Rafal Lukawiecki Strategic Consultant, Project Botticelli Ltd rafal@projectbotticelli.com

33

ANALYZE

INTEGRATE

MANAGE

REPORT

STEAWARDSHIP

Page 4: 1 1 The IT Perspective: Data Warehousing, Management, and Analytical Structures Rafal Lukawiecki Strategic Consultant, Project Botticelli Ltd rafal@projectbotticelli.com

44

SQL Services – Why?Install only the ones you needWhich?

Integration ServicesGet your data from the world outside (ETL)

Analysis ServicesCubes, Data Mining, support for PowerPivot on SharePoint

Reporting ServicesDIY Report Builder and traditional “big” reports

Master Data ServicesQuality of critical master data (cities, colours, customers)

Database EngineData warehouse and OLTP relational storage

Page 5: 1 1 The IT Perspective: Data Warehousing, Management, and Analytical Structures Rafal Lukawiecki Strategic Consultant, Project Botticelli Ltd rafal@projectbotticelli.com

5

Master Data Management

Page 6: 1 1 The IT Perspective: Data Warehousing, Management, and Analytical Structures Rafal Lukawiecki Strategic Consultant, Project Botticelli Ltd rafal@projectbotticelli.com

66

MDM

Ensures consistency of data across all organisational usesImpacts overall data qualityProcesses and tools for:

Collection, aggregation, matching, distribution, and persistence of master data

ConsistentlyRelated to Federated Data Management

Key to MDM: Modelling

Page 7: 1 1 The IT Perspective: Data Warehousing, Management, and Analytical Structures Rafal Lukawiecki Strategic Consultant, Project Botticelli Ltd rafal@projectbotticelli.com

88

MDM Processes

• Batched Acquisition from Staging Tables• Members, Attributes,

Parent-Child Relationships• SQL Integration Services

Import & Integration

• Versioning Changes• Auditing• Compliance• Tracking of Instances

Modeling • Subscription Views• Export to:• Operational Systems• Data Warehouses• BI Analytics• Reporting Tools

Export & Subscription

Page 8: 1 1 The IT Perspective: Data Warehousing, Management, and Analytical Structures Rafal Lukawiecki Strategic Consultant, Project Botticelli Ltd rafal@projectbotticelli.com

99

Microsoft Master Data ServicesSQL 2008 R2 Enterprise, Datacenter, Developer

Tools:Master Data Manager

Primary tool for managing your master data

MDS Configuration Manager

IT Pro toolMDS Web Service

For developers wanting to extend MDS

Concepts:ModelsEntitiesAttributesMembersHierarchiesCollectionsVersionsDatabase

Page 9: 1 1 The IT Perspective: Data Warehousing, Management, and Analytical Structures Rafal Lukawiecki Strategic Consultant, Project Botticelli Ltd rafal@projectbotticelli.com

1010

Modelling Master Data

Model organises data at highest levelAllowing versioning of changes to data

There are typically four categories of models:People (Customers, Staff)Places (Geographies, Cities, Countries)Things (Products)Concepts (Accounts, Behaviours, Transactions)

Page 10: 1 1 The IT Perspective: Data Warehousing, Management, and Analytical Structures Rafal Lukawiecki Strategic Consultant, Project Botticelli Ltd rafal@projectbotticelli.com

1111

Example: Product MDM Model

Product (model)

Product (entity)

Name (free-form attr)

Code (free-form attr)

Subcategory (domain-

based attr)

Name (free-form attr)

Code (free-form attr)

Category (domain-

based attr)

Name (free-form attr)

Code (free-form attr)

StandardCost (free-form

attr)

ListPrice (free-form

attr)

Photo (file attr)

Page 11: 1 1 The IT Perspective: Data Warehousing, Management, and Analytical Structures Rafal Lukawiecki Strategic Consultant, Project Botticelli Ltd rafal@projectbotticelli.com

12

1. Reviewing a Data Model Using Master Data Services

demos

claireh
NOTE TO SPEAKERS: Demo failure is the #1 cause of low scores. Increase speed and reliability by using the virtual demo. For detailed information, consult the Virtual Demo tab of the speaker portal (www.msteched.com).
Page 12: 1 1 The IT Perspective: Data Warehousing, Management, and Analytical Structures Rafal Lukawiecki Strategic Consultant, Project Botticelli Ltd rafal@projectbotticelli.com

13

Data Warehouse

Page 13: 1 1 The IT Perspective: Data Warehousing, Management, and Analytical Structures Rafal Lukawiecki Strategic Consultant, Project Botticelli Ltd rafal@projectbotticelli.com

1414

OLE DB

ODBC

DB2Oracle

XML

SQL Server

Analysis Services

SQL Server

Report Server Models

SQL Server

Data Mining Models

SQL Server

Integration Services

MySAP

Hyperion Essbase

SAP

NetWeaver BISQL

Server

Teradata

Rich ConnectivityData Providers

Page 14: 1 1 The IT Perspective: Data Warehousing, Management, and Analytical Structures Rafal Lukawiecki Strategic Consultant, Project Botticelli Ltd rafal@projectbotticelli.com

1515

Star Schema

Page 15: 1 1 The IT Perspective: Data Warehousing, Management, and Analytical Structures Rafal Lukawiecki Strategic Consultant, Project Botticelli Ltd rafal@projectbotticelli.com

1616

Star Schema Benefits

Simple, not-so-normalized modelHigh-performance queries

Especially with Star Join Query OptimizationMature and widely supportedLow-maintenance

Page 16: 1 1 The IT Perspective: Data Warehousing, Management, and Analytical Structures Rafal Lukawiecki Strategic Consultant, Project Botticelli Ltd rafal@projectbotticelli.com

1717

Snowflake Dimension TablesDefine hierarchies using multiple dimension tablesSupport fact tables with varying granularitySimplify consolidation of heterogeneous data

Potential for slower query performance in relational reporting

No difference in performance in Analysis Services database

Page 17: 1 1 The IT Perspective: Data Warehousing, Management, and Analytical Structures Rafal Lukawiecki Strategic Consultant, Project Botticelli Ltd rafal@projectbotticelli.com

2323

Slowly Changing Dimensions

Maintain historical context as dimension data changesThree common ways (there are more):

Type 1: Overwrite the existing dimension recordType 2: Insert a new ‘versioned’ dimension recordType 3: Track limited history with attributes

Page 18: 1 1 The IT Perspective: Data Warehousing, Management, and Analytical Structures Rafal Lukawiecki Strategic Consultant, Project Botticelli Ltd rafal@projectbotticelli.com

27

Integration and ETL

Page 19: 1 1 The IT Perspective: Data Warehousing, Management, and Analytical Structures Rafal Lukawiecki Strategic Consultant, Project Botticelli Ltd rafal@projectbotticelli.com

2828

Let’s do ETL with SSIS

SQL Server Integration Services (SSIS) serviceSSIS object modelTwo distinct runtime engines:

Control flowData flow

32-bit and 64-bit editions

Page 20: 1 1 The IT Perspective: Data Warehousing, Management, and Analytical Structures Rafal Lukawiecki Strategic Consultant, Project Botticelli Ltd rafal@projectbotticelli.com

3030

Control Flow

Control flow is a process-oriented workflow engineA package contains a single control flowControl flow elements

ContainersTasksPrecedence constraintsVariables

Page 21: 1 1 The IT Perspective: Data Warehousing, Management, and Analytical Structures Rafal Lukawiecki Strategic Consultant, Project Botticelli Ltd rafal@projectbotticelli.com

3131

Data FlowThe Data Flow Task

Performs traditional ETL and moreFast and scalable

Data Flow ComponentsExtract data from SourcesLoad data into DestinationsModify data with Transformations

Service PathsConnect data flow componentsCreate the pipeline

Page 22: 1 1 The IT Perspective: Data Warehousing, Management, and Analytical Structures Rafal Lukawiecki Strategic Consultant, Project Botticelli Ltd rafal@projectbotticelli.com

32

1. Using SQL Server Integration Services for Splitting Data

demos

claireh
NOTE TO SPEAKERS: Demo failure is the #1 cause of low scores. Increase speed and reliability by using the virtual demo. For detailed information, consult the Virtual Demo tab of the speaker portal (www.msteched.com).
Page 23: 1 1 The IT Perspective: Data Warehousing, Management, and Analytical Structures Rafal Lukawiecki Strategic Consultant, Project Botticelli Ltd rafal@projectbotticelli.com

33

OLAP/Multidimensional Data

Page 24: 1 1 The IT Perspective: Data Warehousing, Management, and Analytical Structures Rafal Lukawiecki Strategic Consultant, Project Botticelli Ltd rafal@projectbotticelli.com

3434

Cube = Unified Dimensional Model

Multidimensional dataCombination of measures and dimensions as one conceptual model

Measures are sourced from fact tablesDimensions are sourced from dimension tables

Page 25: 1 1 The IT Perspective: Data Warehousing, Management, and Analytical Structures Rafal Lukawiecki Strategic Consultant, Project Botticelli Ltd rafal@projectbotticelli.com

3737

Hierarchy

Defined in Analysis ServicesOrdered collection of attributes into levelsNavigation path through dimensional spaceVery important to get right!

Customers by Geography

Country

State

City

Customer

Customers by Demographics

Marital

Gender

Customer

Page 26: 1 1 The IT Perspective: Data Warehousing, Management, and Analytical Structures Rafal Lukawiecki Strategic Consultant, Project Botticelli Ltd rafal@projectbotticelli.com

3838

Measure Group

Group of measures with same dimensionalityAnalogous to a fact tableCube can contain more than one measure group

E.g. Sales, Inventory, FinanceDefined by dimension relationships

Page 27: 1 1 The IT Perspective: Data Warehousing, Management, and Analytical Structures Rafal Lukawiecki Strategic Consultant, Project Botticelli Ltd rafal@projectbotticelli.com

3939

Sales Inventory Finance

Customers X

Products X X

Time X X X

Promotions X

Warehouse X

Department X

Account X

Scenario X

Measure Group

Measure GroupD

imen

sio

n

Page 28: 1 1 The IT Perspective: Data Warehousing, Management, and Analytical Structures Rafal Lukawiecki Strategic Consultant, Project Botticelli Ltd rafal@projectbotticelli.com

42

1. Using BIDS to Review Dimension Design

2. Cube Design and Functionality

demos

claireh
NOTE TO SPEAKERS: Demo failure is the #1 cause of low scores. Increase speed and reliability by using the virtual demo. For detailed information, consult the Virtual Demo tab of the speaker portal (www.msteched.com).
Page 29: 1 1 The IT Perspective: Data Warehousing, Management, and Analytical Structures Rafal Lukawiecki Strategic Consultant, Project Botticelli Ltd rafal@projectbotticelli.com

4343

Summary

As a platform for enterprise Business Intelligence you should consider four services:

Data Warehouse (can be relational)Process for Data Management (MDS)Process for Data Integration (ETL)Analysis (OLAP, Data Mining, Columnar)

= SQL Server 2008 R2

Page 30: 1 1 The IT Perspective: Data Warehousing, Management, and Analytical Structures Rafal Lukawiecki Strategic Consultant, Project Botticelli Ltd rafal@projectbotticelli.com

4444

© 2010 Microsoft Corporation & Project Botticelli Ltd. All rights reserved.

The information herein is for informational purposes only and represents the opinions and views of Project Botticelli and/or Rafal Lukawiecki. The material presented is not certain and may vary based on several factors. Microsoft makes no warranties, express, implied or statutory, as to the information in this presentation.

Portions © 2010 Project Botticelli Ltd & entire material © 2010 Microsoft Corp. Some slides contain quotations from copyrighted materials by other authors, as individually attributed or as already covered by Microsoft Copyright ownerships. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Project Botticelli Ltd as of the date of this presentation. Because Project Botticelli & Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft and Project Botticelli cannot guarantee the accuracy of any information provided after the date of this presentation. Project Botticelli makes no warranties, express, implied or statutory, as to the information in this presentation. E&OE.