part 1 - data warehousing lecture at bw cooperative state university (dhbw)

Andreas Buckenhofer

Data Warehouse (Datenbanken II)

Daimler TSS GmbH

Zur Person

Data Warehouse / DHBW / Fall 2016 / Seite 2

https://de.linkedin.com/in/buckenhofer

https://twitter.com/ABuckenhofer

https://www.xing.com/profile/Andreas_Buckenhofer2

Andreas Buckenhofer

Senior DB Professional

[email protected]

Seit 2009 bei Daimler TSS

im Fachgebiet Big Data

Business Unit Analytics

Daimler TSS GmbH

Interner IT-Partner für Daimler

Wir sind Spezialist und strategischer Business-Partner für

innovative IT-Gesamtlösungen im Daimler-Konzern.

Als 100%ige Daimler-Tochter leben wir die Kultur der Exzellenz

mit dem Anspruch der Innovations- und Technologieführerschaft.

Mit herausragender Technologie- und Methodenkompetenz

verstehen wir uns als Anbieter von wettbewerbs-

differenzierenden Dienstleistungen und sind

Impulsgeber in anspruchsvollen IT-Fragestellungen,

speziell in den Kernthemen Car IT und Mobility,

Information Security, Analytics und

Shared Services.


Daimler TSS GmbH

Standorte


Daimler TSS Deutschland

6 Standorte

774 Mitarbeiter

Ulm (Hauptsitz)

Raum Stuttgart

Böblingen, Echterdingen,

Leinfelden, Möhringen

Berlin

Daimler TSS India

Hub Bangalore

19 Mitarbeiter

Daimler TSS China

Hub Beijing

6 Mitarbeiter

Daimler TSS Malaysia

Hub Kuala Lumpur

42 Mitarbeiter

Daimler TSS GmbH Data Warehouse / DHBW / Fall 2016 /

Ganzheitliche Betreuung (Erhebung, Auswertung,

Visualisierung und Interpretation), unabhängige

Beratung und Optimierung der Geschäftsabläufe.

Von klassischer BI bis hin zu predictive und prescriptive

Analytics bieten wir Leistungen unter Berücksichtigung

der Datensicherheit.

Dabei verknüpfen wir fachliche Erfahrung und IT-Know-

how im Daimler-Kontext mit dem Blick fürs große

Ganze.

Analytics. Das große Ganze verstehen, um Daten nutzbar machen.

Daimler TSS GmbH

DWH, Big Data, Data Mining

Data Warehouse / DHBW / Fall 2016 /

• This lecture is about the classical DWH

• 4 sessions

• Mr. Bollinger’s lecture is about Big Data and Data Mining

Daimler TSS GmbH

Overview of the lecture


1. Introduction to DWH, DWH Architectures - 20.10.2016

2. Data Modeling, OLAP 1 - 27.10.2016

3. OLAP 2, ETL - 03.11.2016

4. Metadata, DWH Projects, Advanced Topics - 10.11.2016

Daimler TSS GmbH

About this Lecture


• Structure of the lecture

• Review of the preceding lecture

• Presentation of content

• Group tasks, exercises

• 14:45 – 18:15

• 2x15min breaks

Daimler TSS GmbH

What you will learn today


• Data Warehousing is a major topic of computer science.

• After the end of this lecture you will be able to

• Understand the basic business and technology drivers for data warehousing

• Describe the characteristics of a data warehouse

• Describe the differences between production and data warehouse systems

• Understand DWH architecture(s)

Daimler TSS GmbH

Motivation


• Data as the new oil

• Data as Asset, getting more and more important also in production industries

• Not only classical data-intensive companies like Google or Facebook

• Increasing interest e.g. in insurance, health care, automotive, …

• Connected cars, Smart Home, Tailor-made insurances, etc.

• Hype technologies

• Databases (RDBMS, NoSQL)

• Big Data

• DWH still growing (additional stimuli coming also from Big Data, Analytics, NoSQL, IOT,

RealTime, TimeToMarket, etc)

Daimler TSS GmbH

Many employment opportunities


• DWH department in every (bigger) end user company, also in many medium-sized or

small-sized companies

• DWH department in every (bigger) consulting company

• DWH-only specialized consulting companies

• DWH tool vendors

Daimler TSS GmbH

Many employment opportunities with comprehensive

requirements


• Comprehensive requirements

• DWHs are complex

• Specialists (e.g. Data Integration, Data Visualization, Tool XYZ)

• Generalists (e.g. Data Architects)

Daimler TSS GmbH

Job description examples


Daimler TSS GmbH

• Often used as synonym

• DWH more technical focus

• BI more business / process focus

• “Business intelligence is a set of methodologies, processes, architectures, and

technologies that transform raw data into meaningful and useful information used

to enable more effective strategic, tactical, and operational insights and decision

making.” (Boris Evelson, Forrester Research, 2008)

Data Warehouse (DWH) or Business Intelligence (BI)?


Daimler TSS GmbH

Information Technology (1960‘s – 80‘s)


Support of few business processes in separated applications

• applications separated

• target: replace manual and time consuming activities

• support daily transactions

• data embedded in process / process-specific application

• process-orientation

Daimler TSS GmbH



• Many systems throughout the enterprises for dedicated purposes

• Customer data, order data, etc spread over many systems in many variations and with

contradictions

Daimler TSS GmbH



• Storage of data

• mostly on tape

• very little space, therefore data encoded and compressed

• one separate tape for each process / application

• data spread

• later introduction of databases – however, no consolidation of data throughout the

company

Daimler TSS GmbH



• Databases accessed through common unique keys, numbers, primary keys, ...

• More and more applications access the different databases

• Complex structure of systems and databases, but no overall view of company

possible (silos)

Daimler TSS GmbH



• „Management Information Systems“ (MIS)

• since late 60es

• did not really work

• Can be characterized as “Unplanned decision support”

• Management needs reports / combined data from different systems to make

decisions for company

• Reports all manually written by IT people

• extract, combine, cache, accumulate data

• can take several days

• Error prone

• relevant information may be forgotten or combined in a wrong way

• No ad-hoc questions possible

Daimler TSS GmbH

Major problems for effective decision support


• Distributed data

• Different data structures

• Historic data

• System workload

• Inadequate technology

Daimler TSS GmbH

Distributed data


• Data resides on

• different systems / storages

• different applications

• different technologies

• Has to be accumulated on one system for further analysis

• Data is inhomogeneous, e.g. each system has their own customer number or order

number, etc.

• How to combine the data?

Daimler TSS GmbH

Different data structures


• Systems developed independently from each other

• Different data types

• E.g.; zip-code as integer or character string

• Different encodings

• E.g.; m-f, m-w for gender

• Different field lengths

• E.g.; address field

• Different data modeling

• E.g. last name / first name in different fields vs last name / first name (badly

modelled) in one single field

Daimler TSS GmbH

Issues with historic data


• Usually: Data archived after max. 3 months

• daily transactions produce lots of data

• limited size of storage → high amounts of data fill up systems

• For reports, already archived data may have to be accessed

Daimler TSS GmbH

Issues with system workload


• Systems constructed for daily transaction business

• handle a constant (large) amount of transactions

• handle transactions concurrently

• each transaction accesses small amount of data

• very simple

• Reports:

• few reports at one time

• access lots of data

• not on a regular basis

• complex arithmetic operations

Daimler TSS GmbH

Issues with system workload


• Consequences:

• systems stressed by additional load (due to reports)

• not optimized for this kind of workload

• performance of daily transaction business jeopardized

• may possibly lead to system failure!

Daimler TSS GmbH

Inadequate technology


• Capacity and cost of hard disk drives

• No SAN, inflexible scaling of storage

• E.g. 1979: 8″-Winchester hard disk drive of IBM

• Capacity 5 MB ! Cost 10 000 DM.

• Performance and cost of processors

• No graphical front-ends

• Network technology in its beginnings

Daimler TSS GmbH

Textual and visual Reports


Daimler TSS GmbH

Visual Data Integration Tool


Daimler TSS GmbH

Conclusion


• "operative systems" not suitable for analytical evaluations

• Need for a new, separated system

• fast answers, ad-hoc questions possible

• no interference with daily transaction business

� Data Warehouse

Daimler TSS GmbH

Group Task


• Assume that you are the CIO of a bank, a retail company, a telco provider or an online

store.

• From what you have learned so far list possible (functional and non-functional)

requirements for a data warehouse. Think of the deficiencies of transactional

systems like

• Distributed data

• Different data structures

• Problem with historic data

• Problem with system workload

• Inadequate technology !

• What are requirements from an end-user perspective.

Daimler TSS GmbH

Data Warehouse User


• Wants to access and analyze all important data from desktop

• Uses data analysis as facility for daily work

• Wants data to be fast and dynamically available

• Identifies data correlations by browsing through the data

• does possibly not know what (s)he is looking for

Daimler TSS GmbH

Data Warehouse


• Answer business-critical questions

• Decision support for management

• Overcomes difficulties when using existing transaction systems for those tasks

• Not a product, but a overall concept

• Applications come, applications go. The data, however, lives forever. It is not about

building applications; it really is about the data underneath these application (Tom

Kyte)

Daimler TSS GmbH

Data Warehouse


• Contains data from different systems

• Imports data from different systems on a regular basis

• summarize data

• provide historic data

• generate metadata

• Completely new system

Daimler TSS GmbH

High-Level Data Warehouse Architecture


Staging

OLTP

OLTP

OLTP

Core Warehouse Mart

Data Warehouse

Daimler TSS GmbH

Definitions DWH


Ralph Kimball Bill Inmon

„A data warehouse is a copy of transaction

data specifically structured for querying

and reporting“

“A data warehouse is a subject-oriented,

integrated, time-variant, nonvolatile

collection of data in support of

management’s decision-making process”

Daimler TSS GmbH

• A data warehouse is organized around the major subjects of the enterprise like

• Customer

• Vendor

• Car

• In contrast to the process/functional orientation of applications such as

• Offer

• Booking

• Fulfillment

• Delivery

Subject-oriented


Daimler TSS GmbH

• Data contained in the warehouse are integrated.

• Aspects of integration

• consistent naming conventions,

• consistent measurement of variables,

• consistent encoding structures,

• consistent physical attributes of data

• „One version of truth“

Integrated


Daimler TSS GmbH

• All data in the data warehouse is accurate as of some moment in time.

• Has to be associated with a time stamp.

• In the operational environment data is accurate as of the moment of access.

• Once data is correctly recorded in the data warehouse, it cannot be updated or

deleted.

• Data warehouse data is, for all practical purposes, a long series of snapshots.

• If data is deleted in the OLTP system, the data is the DWH is marked with e.g. a

deletion flag but never actually deleted.

• Operational data, being accurate as of the moment of access, can be updated as the

need arises.

Time-variant


Daimler TSS GmbH

• Operations in operational environment

• insert

• delete

• update

• select

• Operations in a data warehouse

• the initial and additional loading of data by (batch) processes

• the access of data

• (almost) no updates and deletes (technical updates / deletes only)

Nonvolatile


Daimler TSS GmbH

Online Transaction Processing Online Analytical Processing

Transaction-oriented system Query-oriented system

Optimized for insert and update Optimized for complex queries with short

response times; ad-hoc queries

Many users change data Only ETL process writes data

Selective queries on the data Evaluations of all data (complex queries)

Avoid redundancy Redundant data storage

Normalized data management De-normalized data management

Relational Data Modeling Several layers with different data models,

one model usually Dimensional Data

Modeling

OLTP vs OLAP


Daimler TSS GmbH

Operative data Dispositive data

Handling structured, parallel processes

with short and isolated ("atomic")

transactions

information for management (decision

support)

Modeling process- and function oriented,

individual for each application

Different data models in one DWH;

historic, stable and summarized, data

# users many few(er) but increasing user base

System

return time

Milliseconds Seconds to minutes (even hours)

Operative vs dispositive data


Daimler TSS GmbH

Operative DBs Dispositive DBs

Purpose processing of daily business

transactions

information for management (decision

support)

Content detailed, complete, most recent

data

historic, stable and summarized, data

Data

amount

small amount of data per

transaction

large amount of data for load, and often

per query

Data

structure

suitable for operational

transactions

Several models; suitable for long term

storage and business analyses

Trans

actions

ACID; very short read/write

transactions

long load operations, longer read

transactions

Operative vs dispositive databases


Daimler TSS GmbH

• Information oriented

• Analysis oriented

• Campaign oriented

• Planning oriented

Data Warehousing Usage Types


Daimler TSS GmbH

Information oriented


• Computation of various key figures

• Revenue, profit, ..

• Used for „business performance management“

• User groups

• Higher management

• Controlling

• Mostly fixed reports „standard reporting“

• No ad hoc queries

• 70 – 80 % of all users of a data warehouse

• Web-based front-end

• Enterprise Portals

Daimler TSS GmbH

Analysis oriented applications


• Purpose analysis of complex problems

• Example: analysis of quality problems

• Using the information of warranty cases to find underlying reasons for product

failure

• Identification of outliers or quality issues

Daimler TSS GmbH

Campaign oriented applications


• Executed as specific projects

• Campaign planning

• Campaign execution

• Campaign evaluation of results

• Requires

• a broad spectrum of data

• Not all data may be in a data warehouse

• Sophisticated analysis techniques

• Data mining

• Examples

• Marketing campaigns

• Churn prevention

Daimler TSS GmbH

Planning Oriented


• Use data from past to predict trends

• Using trends in the historical data for setting up plan figures

• Futures product sales

• For production planning

• Stock replenishment

• ...

Daimler TSS GmbH

Exercise


• For one of the following companies

• Bank, Telecommunication company, Online store (like Amazon.com), Car

manufacturer, Supermarket

1. Outline the operational systems

• characterize which operations are performed by them

• mostly triggered by actions of a customer

• which information is stored by these systems

• which questions can be answered by these systems

2. For a data warehouse

• describe what information can be stored in it

• which questions can/should be answered with this information

Daimler TSS GmbH

Data Warehouse

FrontendBackend

External data

sources

Internal data

sources

Standard Data Warehouse Architecture


Staging Layer

(Input Layer)

Core Warehouse

Layer

(Storage Layer)

Reporting Layer

(Output Layer)

(Mart Layer)

Daimler TSS GmbH

Data Sources


� Providing internal and external data out of the source systems.

� Enabling data through Push (source is generating extracts) or Pull (BI Data Backend is

requesting or directly accessing data).

� Example for Push practice (deliver csv or text data through file interface; Change

Data Capture (CDC)).

� Example for Pull practice (direct access to the source system via ODBC, JDBC, API

and so on).

Daimler TSS GmbH

Staging Layer


� Major target is to decouple source and target system (repeating extraction run,

additional delivery).

� Granular data (no pre-aggregation or filtering in the Data Source Layer, i.e. the source

system).

� Usually not persistent, therefore regular housekeeping is necessary (for instance after a

few days or weeks or at the latest once a correct upload to Core Warehouse Layer is

ensured).

Daimler TSS GmbH

Core Warehouse Layer


� Data storage in an integrated, consolidated, consistent and often non-redundant

(normalized) data model.

� Application / Reporting neutral data storage on the most detailed level of granularity

(incl. historic data).

� Size of database can be several TB and can grow rapidly due to data historization.

Daimler TSS GmbH

Data Mart Layer


� For performance reasons the Data Mart Layer is providing aggregated data in a

denormalized data model.

� Created through filtering or aggregating the Core Warehouse Layer content on specific

dimensions (e.g. time, region), granularity or business content specific aspects.

� Technically the Data Mart Layer can also be a part of an Analytical Frontend product

(such as QlickTech QlikView or IBM Cognos TM1) and need not to be stored in a

relational database.

� The Data Mart Layer also contains security permissions as end users are granted

access to this layer only.

Daimler TSS GmbH

Other architectures


• Independent Data Marts

• Central data warehouse based on data marts (Kimball approach)

• Virtual Data Warehouse

• Data Vault 2.0 Architecture (Dan Linstedt)

• DW 2.0: The Architecture for the Next Generation of Data Warehousing (Bill Inmon)

Daimler TSS GmbH

Data Warehouse

Frontend

Frontend

Backend

Backend

FrontendBackend

External data

sources

Internal data

sources

Independent Data Marts


Data Mart 1

Data Mart 2

Data Mart 3Staging Layer

(Input Layer)

Core Warehouse

Layer

(Storage Layer)

Daimler TSS GmbH

Independent Data Marts


• Situation is often encountered in enterprises

• Data marts created to solve specific analytical requirements

• Fast implementation and lower cost

• Often Staging Layer and/or Core Warehouse Layer omitted

• Not recommended

• Possible problems

• No integration with other data marts

• No common data model for all data marts

• Cross data mart queries difficult

Daimler TSS GmbH

Data Warehouse

FrontendBackend

External data

sources

Internal data

sources

Central data warehouse based on data marts (Kimball

approach)


Core Warehouse Layer = Mart Layer

Data Mart 1

Data Mart 2Data Mart 3

Staging Layer

(Input Layer)

Daimler TSS GmbH

Central data warehouse based on data marts (Kimball

approach)


• Bottom-up approach

• Leverages existing data marts and integration processes

• Data marts may need to be redesigned if incompatibilities between the data models

exist

• Data mart changes can have an impact on the central data warehouse

• Enterprise service bus / conformed dimensions for integration purposes

Daimler TSS GmbH

Data Warehouse

FrontendBackend

External data

sources

Internal data

sources

Virtual Data Warehouse


Query Management

Daimler TSS GmbH



• Data not extracted from operational systems and stored separately

• Standardized interface for all operational data sources

• One "GUI" for all existing data

• Generates combined queries

• Query Processor joins query result data from different sources

• Can also access data in Hadoop (Polybase, Big SQL, BigData SQL, etc)

Daimler TSS GmbH



• Query Management manages metadata about all operational systems

• (physical) location of data

• Algorithms for extracting data from operational system

• Implementation easier

• Low cost: can use existing hardware infrastructure

• Optimized access to existing infrastructure

Daimler TSS GmbH



• Queries cause significant performance problems in operational systems

• Known problems when analyzing operational data directly

• Same query is processed multiple times (if queried multiple times)

• Same query delivers different results when processed at different times

Daimler TSS GmbH

Data Warehouse

FrontendBackend

External data

sources

Internal data

sources

Data Vault 2.0 Architecture (Dan Linstedt)


Staging Layer

(Input Layer)

Core Warehouse

Layer

(Storage Layer)

Reporting Layer

(Output Layer)

(Mart Layer)

Daimler TSS GmbH

Data Vault 2.0 Architecture


• Core Warehouse Layer is modeled with Data Vault

• Business rules are applied from Core Warehouse Layer to Mart Layer and not earlier

• Or alternatively within Core Warehouse Layer (Raw Data Vault containing data

without transformations and Business Data Vault containing transformed data)

• Data is fully auditable

• Real-time capable architecture

Daimler TSS GmbH

Data Vault 2.0 Architecture


• Data Vault is optimized for the following requirements:

• Flexibility

• Agility

• Data historization

• Data integration

• Auditability

• Bill Inmon wrote in 2008: “Data Vault is the optimal approach for modeling the EDW in

the DW2.0 framework.” (DW2.0)

Daimler TSS GmbH

DW 2.0: The Architecture for the Next Generation of Data

Warehousing (Bill Inmon)


Daimler TSS GmbH

DW 2.0: The Architecture for the Next Generation of Data

Warehousing (Bill Inmon)


• Structured and unstructured data

• Different storage areas

• Hot data: High speed, expensive storage (RAM, SSDs) for most recent data

• …

• Cold data: Low speed, inexpensive storage (e.g. hard disks) for old data

Daimler TSS GmbH

Architecture from an actual Project


ETL Engine

Fro

nte

nd

Standard

Reports

AdHoc

ReportsLogs

TSM

IIDR

ReplEngine

Source

Datastore

Source

Spiegel

DB

OLTP

DB

IIDR ReplEngine

Spiegel

Datastore

Spiegel

IIDR ReplEngine

DWH

Datastore

DWH

DWH DB

Staging Layer

Core Layer / Data Vault

Mart Layer

Daimler TSS GmbH

Group Task


• Review the presented data warehouse architectures.

• Which architecture would you recommend for

• A multinational telco provider

• A holding of 3 car manufacturers

• An online store represented in almost all countries

• A machine tools manufacturer with plants in Europe, America and Asia

• You may recommend a combination of the architecures or you may recommend as well

different architectures for different subject areas

• List advantages and drawbacks of your proposal.

Thank you!

Daimler TSS GmbH

Wilhelm-Runge-Straße 11, 89081 Ulm, Germany / Phone +49 731 505-06 / Fax +49 731 505-65 99

[email protected] / Internet: www.daimler-tss.com / Intranet portal code: @TSS

Domicile and Court of Registry: Ulm / Commercial Register No.: 3844 / Management: Christoph Röger (Vorsitzender), Steffen Bäuerle

part 1 - data warehousing lecture at bw cooperative state university (dhbw)

Data & Analytics