part 3 - data warehousing lecture at bw cooperative state university (dhbw)

Andreas Buckenhofer

Data Warehouse (Datenbanken II)

Daimler TSS GmbH

Overview of the lecture

Data Warehouse / DHBW / Fall 2016 /

1. Introduction to DWH, DWH Architectures - 20.10.2016

2. Data Modeling, OLAP 1 - 27.10.2016

3. OLAP 2, ETL - 03.11.2016

4. Metadata, DWH Projects, Advanced Topics - 10.11.2016

Daimler TSS GmbH

What you will learn today


• After the end of this lecture you will be able to

• Understand advanced topics of OLAP

• Understand data integration

• ETL

• Data quality

Daimler TSS GmbH

OLAP


Daimler TSS GmbH

How to cover data changes?


• Data changes, e.g.

• new employees

• employees change departments

• employees leave

• whole department reorganisations, etc

• How are the changes handled?

• What does the business want to see? (Reporting Scenarios)

• How is data inserted / updated in dimensions? (Slowly Changing Dimensions)

Daimler TSS GmbH

Reporting scenarios


• As-is scenario

• As-of scenario

• As-posted scenario

• As-posted with comparable data scenario

Daimler TSS GmbH

Data Mart – example baseline


Employee Organisation

Miller DWH

Rogers DWH

Douglas Database

Powell Database

Em

plo

yee

Dim

en

sio

n 2

015


Miller DWH

Rogers DWH

Powell DWH

Douglas Database

Bush Database

Em

plo

yee

Dim

en

sio

n 2

016

Employee Year #Pro-

jects

Miller 2015 10

Rogers 2015 10

Douglas 2015 10

Powell 2015 10

Miller 2016 10

Rogers 2016 10

Powell 2016 10

Douglas 2016 10

Bush 2016 10Facts

Daimler TSS GmbH

As-is scenario


Reporting uses current structure


Miller DWH

Rogers DWH

Powell DWH

Douglas Database

Bush Database

Em

plo

yee

Dim

en

sio

n 2

016

Employee Year #Pro-

jects

Miller 2015 10

Rogers 2015 10

Douglas 2015 10

Powell 2015 10

Miller 2016 10

Rogers 2016 10

Powell 2016 10

Douglas 2016 10

Bush 2016 10Facts

Organisation #Projects ´15 #Projects ´16

DWH 30 30

Database 10 20

Daimler TSS GmbH

As-of scenario


Reporting uses structure as demanded


Miller DWH

Rogers DWH

Douglas Database

Powell Database

Em

plo

yee

Dim

en

sio

n 2

015

Employee Year #Pro-

jects

Miller 2015 10

Rogers 2015 10

Douglas 2015 10

Powell 2015 10

Miller 2016 10

Rogers 2016 10

Powell 2016 10

Douglas 2016 10

Bush 2016 10Facts


DWH 20 20

Database 20 20

Daimler TSS GmbH

As-posted scenario


Reporting uses „historical truth“Employee Year #Pro-

jects

Miller 2015 10

Rogers 2015 10

Douglas 2015 10

Powell 2015 10

Miller 2016 10

Rogers 2016 10

Powell 2016 10

Douglas 2016 10

Bush 2016 10Facts


DWH 20 30

Database 20 20

Daimler TSS GmbH

As-posted with comparable data scenario


Reporting uses „historical truth“Employee Year #Pro-

jects

Miller 2015 10

Rogers 2015 10

Douglas 2015 10

Powell 2015 10

Miller 2016 10

Rogers 2016 10

Powell 2016 10

Douglas 2016 10

Bush 2016 10Facts


DWH 20 20

Database 10 10

Daimler TSS GmbH

Slowly changing dimensions


• Dimensions must absorb changes

• Slowly changing dimensions according to Kimball / Ross (2002):

• SCD Type 0

• no changes, new data is ignored

• SCD Type 1

• See next slides

• SCD Type 2

• See next slides

• SCD Type 3

• See next slides

• And some more SCD types

• Rarely relevant

Daimler TSS GmbH

Slowly changing dimensions – example baseline


• Changes:

• New data added: Albert, DWH

• Powell marries and has new name Parker

ID Employee Organisation

1 Miller DWH

2 Powell Database

Em

plo

yee

Dim

en

sio

n

Daimler TSS GmbH

Slowly Changing Dimension Type 1


• No History

• Dimension attributes always contain current data

Em

plo

yee

Dim

en

sio

n


1 Miller DWH

2 Parker Database

3 Albert DWH

Em

plo

yee

Dim

en

sio

n

• Changes:




1 Miller DWH

2 Powell Database

Daimler TSS GmbH



• Historization of latest change only

• And storage of current value

Em

plo

yee

Dim

en

sio

n

ID Employee Name Previous Name Organisation Previous

Organisation

1 Miller NULL DWH NULL

2 Parker Powell Database NULL

3 Albert NULL DWH NULL

Em

plo

yee

Dim

en

sio

n

• Changes:




1 Miller DWH

2 Powell Database

Daimler TSS GmbH



• Full Historization

• Dimension contains timestamps

Em

plo

yee

Dim

en

sio

n

ID Employee Organisation Valid From Valid To

1 Miller DWH 01.01.2015 NULL (or 31.12.9999)

2 Powell Database 21.12.2014 15.10.2016

3 Albert DWH 05.03.2014 NULL (or 31.12.9999)

2 Parker Database 15.10.2016 NULL (or 31.12.9999)

Em

plo

yee

Dim

en

sio

n

• Changes:




1 Miller DWH

2 Powell Database

Daimler TSS GmbH

Dimension types: Conformed dimension


• Dimension that is used in several fact tables

• Fact tables can be connected by using conformed dimensions

Sales

Fact

Inventory

Fact

Product

Dimension

Location

Dimension

Daimler TSS GmbH

Dimension types: Conformed dimension


• Kimball: Enterprise DWH Bus Matrix is a “design tool” to document the

organization’s processes

Date Product Location Customer Promotion

Sales Fact X X X X X

Inventory Fact X X X

Customer

Returns Fact

X X X X

Sales Forecast

Fact

X X X

Daimler TSS GmbH

Dimension types: Junk dimension


• Collection of random codes that could also form it’s own dimension

ID MartialStatus Gender

1 Single Male

2 Single Female

3 Married Male

4 Married Female

Daimler TSS GmbH

Dimension types: Role-playing dimension


• A single dimension is referenced several times by a fact table

• E.g. several dates in fact table reference Date Dimension

ID OrderDate DeliveryDate ProductionDate

1 .. .. ..

2 .. .. ..

3 .. .. ..

4 .. .. ..

Daimler TSS GmbH

Dimension types: Degenerated dimension


• A dimension without own dimension table. Data are stored in the fact table only.

• Used e.g. for drill-through in reports

• E.g. OrderNumber in sales fact table

ID OrderNumber

1 A51273 .. ..

2 72841 .. ..

3 732GT5 .. ..

4 624TR5K .. ..

Daimler TSS GmbH

Temporal data storage (Bitemporal data)


10.09. 20.09. 30.09. 10.10.

Time

Price: 15EUR Price: 16EUR

New Price of 16EUR is

entered into the DB

Valid

Time

(20.09.)

Transaction

Time

(10.09.)

Daimler TSS GmbH



• Valid time is the time period during which a fact is true in the real world.

• Transaction time is the time period during which a fact stored in the database was

known.

• Bitemporal data combines both Valid and Transaction Time.

• Source: (Wikipedia, https://en.wikipedia.org/wiki/Temporal_database)

Daimler TSS GmbH



• SQL standard SQL:2011

• But different implementations by RDBMSes like Oracle, DB2, SQL Server and others

• Different syntax!

• Different coverage of standard!

• Very useful for slowly changing dimensions type 2, but also for other purposes

Daimler TSS GmbH

DB2 Valid Time example


CREATE TABLE customer_address

( customerID INTEGER NOT NULL

, name VARCHAR(100)

, city VARCHAR(100)

, valid_start DATE NOT NULL

, valid_end DATE NOT NULL

, PERIOD BUSINESS_TIME(valid_start, valid_end)

, PRIMARY KEY(customerID, BUSINESS_TIME WITHOUT OVERLAPS)

);

Daimler TSS GmbH



INSERT INTO customer_address VALUES

(1, 'Miller', 'Seattle', '01.01.2013', '31.12.2013');

UPDATE customer_address

FOR PORTION OF BUSINESS_TIME

FROM '22.05.2013' TO '31.12.2013'

SET city = 'San Diego'

WHERE customerID = 1;

customerID Name City Valid_start Valid_end

1 Miller Seattle 01.01.2013 22.05.2013

1 Miller San Diego 22.05.2013 31.12.2013

Daimler TSS GmbH



SELECT *

FROM customer_address

FOR BUSINESS_TIME AS OF '17.05.2013';

Daimler TSS GmbH

DB2 Transaction Time example


CREATE TABLE customer_info(

customerId INTEGER NOT NULL,

comment VARCHAR(1000) NOT NULL,

sys_start TIMESTAMP(12) NOT NULL GENERATED ALWAYS AS

ROW BEGIN,

sys_end TIMESTAMP(12) NOT NULL GENERATED ALWAYS AS

ROW END,

PERIOD SYSTEM_TIME (sys_start, sys_end)

);

Daimler TSS GmbH



Transaction on 15.10.2013:

INSERT INTO customer_info VALUES(

1, 'comment 1'

);

Transaction on 31.10.2013

UPDATE customer_address SET comment = 'comment 2‘

WHERE customerID = 1;

CustomerId comment Sys_start Sys_end

1 Comment 2 31.10.2013 31.12.2999

Daimler TSS GmbH



SELECT *

FROM customer_info FOR SYSTEM_TIME AS OF '17.10.2013';

Data comes from a history table:

Valid Time and Transaction Time can be combined = Bitemporal table

CustomerId comment Sys_start Sys_end

1 Comment 1 15.10.2013 31.12.2999

Daimler TSS GmbH

Hierarchies – non-normalized hierarchy table


Daimler TSS GmbH

Hierarchies


� Hierarchy data in one or more tables

� the dimension table(s)

� MOLAP: Aggregated values for each hierarchy level stored

� ROLAP: Aggregated values dynamically calculated

� i.e. through SQL built-in aggregation functions

� Storage of aggregated data in ROLAP:

� Fact data record with aggregated data

� Materialized view/query table

Daimler TSS GmbH

Aggregations - Types of facts/measures


• Additive

• Can be summed up through all of the dimensions in the fact table.

• Example: Retail data warehouse:

• Dimensions: time, location, customer and product

• Measure: sales amount

• Semi-Additive

• Can be summed up for some of the dimensions in the fact table only.

• Example: Banking data warehouse

• Dimensions: time, account

• Measure: current balance

Daimler TSS GmbH

Aggregations - Types of facts/measures


• Non-Additive:

• Cannot be summed up for any of the dimensions present in the fact table

• Example: Retail data warehouse:

• Dimensions: time, location, customer and product

• Measure: ratios

• Can be computed from additive or semi-additive facts

Daimler TSS GmbH

Aggregation function


• Have to be defined for each measure and dimension

• Sum is the most frequently used

• Other possible aggregation function

• count

• average

Daimler TSS GmbH

Types of fact tables


• Transactional

• Most common

• Usually one row per line/event in a transaction

• Most detailed level

• The grain must (should) be the same for all rows

• E.g. sales data

• Periodic snapshots

• Picture of the time

• Often computed from transactional fact table, e.g. aggregated by month


• E.g. inventory data (summed up for each day)

Daimler TSS GmbH

Types of fact tables


• Accumulating snapshots

• Shows activity of a process/event over time

• The data is not complete at the beginning and is updated as soon as new data

arrived (e.g. delivery date can be unknown at the beginning)


• E.g. processing an order

Daimler TSS GmbH

MDX - OLAP Query Language


• ROLAP = SQL is standard language

• MOLAP = MDX - Multidimensional Expressions

• De-facto industry standard developed by Microsoft

• Very complex

• SQL like syntax

• Language elements

• Scalar – data type „string“ or „number“

• Dimension

• Hierarchy

• Level

• Member

• …

Daimler TSS GmbH

MDX Sample Query


SELECT

{ [Measures].[Store Sales] } ON COLUMNS,

{ [Date].[2002], [Date].[2003] } ON ROWS

FROM Sales

WHERE ( [Store].[USA].[CA] )

• This query defines the following result set information:

• The SELECT clause sets the query axes as the Store Sales (amount) member and

the 2002 and 2003 members of the Date dimension.

• The FROM clause indicates that the data source is the Sales cube.

• The WHERE clause defines the "slicer axis" as the California member of the Store

dimension.

Store Sales

2002 95863,66

2003 99764,01

Daimler TSS GmbH

OLAP Engines


• Middleware between

• Reporting/BI Frontend tool and

• (relational or multidimensional) data store

• Provide a logical multidimensional view on OLAP cubes independently of their storage

scheme

• Holds OLAP metadata (dimensions, hierarchies, measures, ..)

• Usually support MDX through corresponding application programming interfaces

• ODBO - OLE DB for OLAP

• XMLA – XML for Analysis

• E.g. IBM Cognos TM1, Oracle Essbase, Microsoft Analysis Services, Oracle OLAP

Option, IBM Cognos Powerplay

Daimler TSS GmbH

Sample OLAP Engine: Cubing Services in IBM InfoSphere

Warehouse 9.5.x


Daimler TSS GmbH

Cube Server in Action – Startup


Daimler TSS GmbH

Cube Server in Action – Query Processing


Daimler TSS GmbH

ROLAP: Cognos Report Studio example


Daimler TSS GmbH

Exercise


• We designed two data models in the last session

• Data Vault

• Star Schema

• The customer in the business department has additional requirements:

• An engine must be added to both data model. An engine has a unique identification

number

• Cars can have several engines types nowadays at the same time, “classic” engine +

electric engine. Assume that it is sufficient to reference 2 engines at the same time

• Cars can have several engines over time, e.g. an engine is replaced because of a

defect

• Enlarge both data models with the new requirements

• What happens to existing data if the model is enlarged? Is a migration necessary?

Daimler TSS GmbH

Exercise


Daimler TSS GmbH

Exercise Data Vault, possible solution


Daimler TSS GmbH

Exercise Star Schema, possible solution


Daimler TSS GmbH

Exercise: What happens to existing data if the model is

enlarged? Is a migration necessary?


• Data Vault:

• Additional tables only, no reload of data necessary or other changes

• Star Schema

• Migration necessary

• Changes in Fact table (new fields / foreign keys)

• Existing data must be changed (updated)

• usually reloaded as updates on high amounts of data is slow

• Existing code to load data must be changed

Daimler TSS GmbH

ETL – Extract, Transform, Load


Daimler TSS GmbH

Data Warehouse

FrontendBackend

External data

sources

Internal data

sources

Standard Data Warehouse Architecture


Staging Layer

(Input Layer)

Core Warehouse

Layer

(Storage Layer)

Reporting Layer

(Output Layer)

(Mart Layer)

? ? ?

Daimler TSS GmbH

ETL Process


• Monitor changes in source systems

• Other term: Data integration

Daimler TSS GmbH

Tasks of the ETL Process


• Extract

• capture and copy data from source systems (e.g. operational systems)

• many different types of sources

• Relational, hierarchical DBMSs

• Flat files

• Other internal/external sources

• Transform

• Filter data

• Integrate data

• Check and cleanse data

• Load

• Fast load into staging area or another layer

Daimler TSS GmbH

ETL vs ELT


Extract – Transform – Load

• ETL often used for data integration in general (for ETL and ELT)

• But:

• ELT is differentiated from ETL

Source

DB

Target

DB

ETL Server

Source

DB

Target

DB

ELT Server

Datenfluss

Daimler TSS GmbH

ETL vs ELT


ETL ELT

Data is transferred to ETL server and transferred

back to DB. High network bandwidth required

Data remains in the DB except for cross

Database loads (e.g. source to target)

Transformations are performed in the ETL Server Transformations are performed (in the source or)

in the target

Proprietary code is executed in the ETL server Generated code, e.g. SQL, PL/SQL, SQLT

Typically used for

• source to target transfer

• Compute intensive transformations

• Small amount of data

Typically used for

• High amounts of data

Daimler TSS GmbH

ETL Tool vs manual ETL


ETL Tool Manual ETL

Informatica, Talend, Oracle ODI, etc. SQL, PL/SQL, SQLT, etc.

Separate license No additional license

Workflow, error handling, and restart/recovery

functionality included

Workflow, error handling, and restart/recovery

functionality must be implemented manually

Impact analysis and where-used (lineage)

functionality available

Impact analysis and where-used (lineage)

functionality difficult

Faster development, easier maintenance Slower development, more difficult maintenance

Additional (Tool-) Know How required Know How often available

Daimler TSS GmbH

ETL Server


Extract

services

Load

services

Operations management services

Scheduler Control Repository Management

Connectors

Sorter

Connector

Sorter

Bulk Loader

Data Profiling servicesSource analysis

Data Quality servicesData cleansing

Data Transformation and Integration services

Data mapping Business rules

Slowly Changing Dimensions

Datatype conversion

Lookups

Job Monitoring Auditing Error Handling

Security

Daimler TSS GmbH

Mapping - Informatica


Source TargetFilter

Lookup

Daimler TSS GmbH

Mapping with Transformations - Informatica


SorterAggregator

Transformation

Union

Transformation

Daimler TSS GmbH

Workflow - Informatica


Session containing

Mapping

Decision &

coordination step

Session containing

Mapping

Session containing

Mapping

Daimler TSS GmbH

Job Monitoring - Informatica


Daimler TSS GmbH

Monitoring (Data change detection)


• Extracts from source systems

• Initial extract for setting up the data warehouse

• Initial Load

• Periodical extracts for adding new/changed information to the data warehouse

• Incremental Load

• Question: How to determine what is new or what has changed in the source systems?

� Task of „monitoring“

Daimler TSS GmbH

Monitoring: net effect of changes


• Discovery of all changes vs. determining the net effect at extract/load time only

• Example: an attribute value can be changed in two ways:

• by one update operation

• by one delete and one insert operation

• The net effect of both is the same

• However, history information is lost if the net effect is recorded only

Daimler TSS GmbH

Exercise


• Which techniques can be used to identify changes in a source system (RDBMS)?

• E.g. in OLTP system

• new products are inserted

• customer address changes

• Product is deleted because it is out of stock

• How would you identify such changes? List advantages / disadvantages of possible

solutions

• Think about making changes in the source system. Think also about other solutions

without any change in the source system.

Daimler TSS GmbH

Monitoring techniques


• Depend on characteristics of the data sources

• The following techniques are based on modern relational DBMS

• Types of techniques

• Based on DBMS

• Trigger-based

• Log-based discovery

• Replication techniques

• Controlled by application

• Timestamp-based discovery

• Snapshot-based discovery

Daimler TSS GmbH

Trigger-based


• Active monitoring mechanisms

• Based on (database) triggers

• Example:

• If new record is inserted in sales transaction table then insert transaction id

and timestamp in change table

• Advantage:

• Triggers do not change operational applications

• Disadvantage:

• Performance impact on operation systems if triggers are used extensively

• Triggers have to be implemented for every table in the source systems

Daimler TSS GmbH

Trigger-based


• Sample Trigger Code, Oracle

CREATE [OR REPLACE] TRIGGER <trigger_name>

{BEFORE|AFTER} {INSERT|DELETE|UPDATE}

ON <table_name>

[REFERENCING [NEW AS <new_row_name>] [OLD AS

<old_row_name>]]

[FOR EACH ROW [WHEN (<trigger_condition>)]]

<trigger_body>

• Trigger is created for each source table in OLTP DB and stores insert/update/delete

changes in a “log/journal table”

• trigger body contains insert statements into log/journal table

Daimler TSS GmbH

Log-based


• Log-based discovery

• Also known as CDC (Change Data Capture)

• Usage of database logs to determine changes

• DBMSs write transaction logs in order to be able to undo partially executed

transactions

• This information can be used to determine all changes

• Log reader identifies insert, update, delete, truncates and writes the changes as

inserts into staging layer

• Transaction Log files can be transferred to other systems to avoid additional load on

source systems

Daimler TSS GmbH

Log-based (sample product architecture IIDR)


Fro

nte

nd

Standard

Reports

AdHoc

ReportsLogs

IIDR

ReplEngine

Source

Datastore

Source

OLTP

DBIIDR ReplEngine

DWH

Datastore

DWH

DWH DB

Staging Layer

Core Layer

Mart Layer

Daimler TSS GmbH

Replication-based


• Replication techniques

• Data replication

• Target tables not necessarily on local system

• Uses typically Transaction Logs

• Log reader identifies insert, update, delete, truncates and writes the changes into

replicated tables (insert remains insert, update remains update, etc)

• Useful for 1:1 copies but still challenge to detect changes for loading the data

mart

Daimler TSS GmbH

Replication-based (sample product architecture IIDR)


Fro

nte

nd

Standard

Reports

AdHoc

ReportsLogs

IIDR

ReplEngine

Source

Datastore

Source

OLTP

DB

IIDR ReplEngine

Spiegel

Datastore

Spiegel

DWH DB

Staging Layer

Core Layer

Mart Layer

Mirror

DB

Daimler TSS GmbH

Timestamp-based


• Timestamp-based discovery

• Every data item in a table is associated with timestamp information about its validity

period

• Changed data can be determined from this timestamp information

• Operational applications have to keep a limited change history

Daimler TSS GmbH

Timestamp-based


• Sample customer table in OLTP

• Each table gets Change timestamp

• Delta process reads latest data only (e.g. ChangeTimestamp >= <yesterday>)

• Problem: it is not possible to identify deleted rows

CustomerID Name Department Change Timestamp

1 Miller DWH 15.01.2015 17:00:01

2 Powell DB 22.03.2016 08:30:22

Daimler TSS GmbH

Snapshot-based


• Data comparison

• Comparison of snapshots of the operational data at different points in time

• Compute difference between two latest snapshots

• E.g. unload all data from a table into a file and diff newest file content with latest

file content

• Can be very complex

• Sometimes the only possibility, for instance for legacy applications

• High performance impact on source

Daimler TSS GmbH

Monitoring techniques comparison


Trigger-based Replication

techniques

Log-based

discovery

Timestamp-

based

discovery

Snapshot-

based

discovery

Performance

impact on

source system

Medium Low Low Medium High

Performance

impact on

target system

Low Low Low Low High

Load on

network

Low Low Low Low High

Dataloss if

nologging

operations

No Yes Yes No No

Daimler TSS GmbH

Monitoring techniques comparison


Trigger-based Replication

techniques

Log-based

discovery

Timestamp-

based

discovery

Snapshot-

based

discovery

Identify

DELETE

operations

Yes Yes Yes No Yes

Identify ALL

changes

(changes

between

extractions)

Yes Yes Yes No No

Near Real-Time

ready

Maybe Yes Yes Unlikely Unlikely

Daimler TSS GmbH

Data Transport


• Direct Access

• Source writes data into target or

• Target reads data from source

• Security concerns

• High coupling / dependencies

• File transfer (or other transport medium)

• csv, json, xml, binary, etc

• Transfer data by scp, rfts (reliable file transfer system), ESB (enterprise service bus),

SOA (service oriented architecture), etc

• Often high amounts of data, therefore bulk transfer of compressed data most widely

used

• Better decoupling of source and target

Source Target

Source Target

Daimler TSS GmbH

Extraction intervals


• Extraction intervals

• Periodically – in regular intervals

• Every day, week, etc.

• Instantly / Continuous

• Every change is directly propagated into the data warehouse

• „real time data warehouse“

• Depends on the requirements on timeliness of the data warehouse data

• Triggered by a specific request

• Addition of a new product

• Query which involves more recent data

• Triggered by specific events

• Number of changes in operational data exceeds threshold

Daimler TSS GmbH

Prerequisite of ETL - Understanding The Data


• Profile Existing Data Sources, Extracted Data

• Analyze data structure, content, and quality

• Find data relationships across systems

• Often badly documented or missing foreign keys

• Uncover data issues that can affect subsequent transformation steps

• Missing values

• Duplicates

• Inconsistencies

Daimler TSS GmbH

Exercise


• For one of the following companies

• Bank

• Telecommunication company

• Online book store (like Amazon.com)

• Supermarket

describe 5 potential data quality problems.

• What could be done to prevent these problems?

• Which impact might these problems have on its business?

Daimler TSS GmbH

Data Quality issues


CustomerNo Name Birthdate Age Gender Zip code

1 Miller, Tom 33.01.2001 15 M NULL

1 John Mayor 15.01.2001 15 M 98144

2 Mrs. Bush 31.10.1988 22 Q 00000

PK / Unique Key

violatedData not uniform Not valid

Inconsistent Wrong value

Unknown / missing

FK violated

Daimler TSS GmbH

Issue Solution

Wrong data e.g. 31.02.2016 Proper data type definition

Wrong values, e.g. number out of range Check constraint

Missing values NOT NULL constraint

Violated references FOREIGN KEY constraint

Duplicates PRIMARY or UNIQUE KEY constraint

Inconsistent data ACID transactions, business logic, additional checks

Data Quality issues: solutions (prevention) in OLTP


Daimler TSS GmbH

Data Quality issues: workarounds in DWH


• Correcting the data

• Automatically during ETL

• E.g., address of a customer if a correct reference table exists

• Manually after ETL is finished

• ETL stored bad data in error log tables or files

• ETL flags bad data (e.g. invalid)

• At the source systems

• Common master data management across all operational applications

• Dedicated systems are “master” of e.g. customer data

• Correcting the data at the source is best approach but slow and often not

feasible

Daimler TSS GmbH

Data Quality issues: missing data


• Column is null

• Reject data

• Use default values

• Missing values can represent

• an unknown value

• Iike date of birth of a customer

• a value that does not exist

• like „engine type“ for bicycle in a vehicles table

• Dimension tables can include some dummy values:

DimensionTable_X Description

-1 Unknown

-2 Missing

Daimler TSS GmbH

Data Quality issues: dirty data


• Data is inaccurate, e.g. wrong date 32.12.2015 or wrong number 55U

• Reject data

• Replace with value that represents „Invalid“

• Dimension tables can include some dummy values:

DimensionTable_X Description

-1 Unknown

-2 Missing

-3 Invalid

Daimler TSS GmbH

Data Quality issues: conflicting data


• Data has conflicts, e.g. wrong postal code 80995 Stuttgart

• Reject data

• Replace one of the values with a value that represents „Invalid“

• Which value to replace? Rules necessary

Daimler TSS GmbH

Data Quality issues: inconsistent data


• Data is inconsistent, e.g. Order date after payment date or unlikely high price for a

product

• Can be discovered by statistical and data mining methods

Daimler TSS GmbH

Data Quality issues: duplicates


• Data is duplicated, e.g. „Martin Miller” vs “Miller, Martin” vs “M.Miller”

• Multiple representations for one entity

• Keys

• Encodings

• Duplicate detection can be very difficult / tricky

• Standardize / Harmonize data during ETL flow: “unification” for better duplicate

detection

Daimler TSS GmbH

Transform - Unification of data


• Unification of data types

• Character string � date „20.01.2006“ � 20.01.2006

• Character string � number „12345“ � 12345

• Unification of encodings

• For instance for gender F and M

• Lookup-tables contain the mapping from old to new encodings

• Unification of names:

• „last name“, „first name“ like „Maier, Peter“

� separate into “Peter” and “Maier”

• Can become very challenging “Herr Prof. Dr. Hans M. vom und zum Stein” or

“Werner Martin” or “Mariae Gloria … Wilhelmine Huberta Gräfin von Schönburg-

Glauchau“

Daimler TSS GmbH



• Unification of dates and timestamps

• Rules for representing incomplete date information

• If only month and year are known

• Dates and timestamps with regard to one specific timezone

• Important for multi-national organizations

• UTC Coordinated Universal Time without daylight saving zone

• Combination of different attributes to one attributes

• day, month, year � date

• Split of one attribute into two or more

• Name � first name, last name (“Herr Prof. Dr. Hans M. vom und zum Stein”)

• Product name - „Cola, 0.33 l“

� Product short name - „Cola“, size in liters - 0.33

Daimler TSS GmbH



• Computation of derived values

• Profit = sales price – purchase price

• Without clear definition, different interpretations possible

• Net or gross sales price?

• Net or gross purchase price?

• Aggregations

• Revenue of the year computed from revenues of the day

• Without clear definition, different interpretations possible

• Calendar year?

• Fiscal year?

Daimler TSS GmbH

Data Mapping


• Specification between source and target columns

• Source tables + columns

• Target table + columns

• Join rules

• Filter criteria

• Transformation rules

Daimler TSS GmbH

Load


• Efficient load operations are important

• bulk load or bulk processing in general

• Single row processing vs set based processing

• Online load

• Data warehouse (especially Data Mart) is still accessible

• For incremental updates

• Offline load

• Data warehouse (especially Data Mart) is offline

• For updates that require the recomputation of a cube

• Offline load is often a Tool limit because the Tool locks data structures

• Because of such locks, the offline load is normally faster. Therefore the Offline

load is often run instead of an online load if there were many data changes

Daimler TSS GmbH

Bulk processing


• Specific Bulk load operations provided by RDBMS, e.g. External tables in Oracle or

LOAD command in DB2

• Single row vs set based processing

Single row processing Set based processing

Cursor curs = SELECT * FROM <source>

WHILE NOT EOF(curs)

FETCH NEXT ROW INTO myRoW;

INSERT INTO <target> VALUES(myRow);

LOOP

INSERT INTO <target>

SELECT * from <source>

Error handling easy All or nothing if there are errors

Slow for high amounts of data Performs well for small and high amounts of data

More coding Less code = less errors

Daimler TSS GmbH

ETL-Job parallelism for loading data into Core Warehouse

Layer

Data Warehouse / DHBW / Fall 2016 / Seite 96

HU

B lo

ad

ed

LIN

K u

nd

HU

B-

SA

Tlo

ad

ed

LIN

K-S

AT

load

ed

Da

ta V

au

lt

Lo

ad

Cla

ssic

al

Lo

ad

?

? ?

Integration of new JobsTime Windows for Loads, e.g 00:00-06:00

• Complex

• Many dependencies

• Many sequential jobs

• Systematic / Methodic

• Few, well defined dependencies

• Massive parallel

Daimler TSS GmbH

Data Warehouse

FrontendBackend

External data

sources

Internal data

sources

Standard Data Warehouse Architecture


Staging Layer

(Input Layer)

Core Warehouse

Layer

(Storage Layer)

Reporting Layer

(Output Layer)

(Mart Layer)

? ? ?

Thank you!

Daimler TSS GmbH

Wilhelm-Runge-Straße 11, 89081 Ulm, Germany / Phone +49 731 505-06 / Fax +49 731 505-65 99

[email protected] / Internet: www.daimler-tss.com / Intranet portal code: @TSS

Domicile and Court of Registry: Ulm / Commercial Register No.: 3844 / Management: Christoph Röger (Vorsitzender), Steffen Bäuerle

part 3 - data warehousing lecture at bw cooperative state university (dhbw)

Data & Analytics