irm uk - 2009: dv modeling and methodology

04/10/2023LearnDataVault.com 1

Data Vault Modeling

MethodologyA Primer…

© Dan Linstedt 2009-2012All Rights Reserved

http://LearnDataVault.com

3

A bit about me…

• Author, Inventor, Speaker – and part time photographer…

• 25+ years in the IT industry• Worked in DoD, US Gov’t, Fortune 50, and

so on…

• Find out more about the Data Vault:o http://www.youtube.com/LearnDataVaulto http://LearnDataVault.com

• Full profile on http://www.LinkedIn.com/dlinstedt

LearnDataVault.com

http://www.youtube.com/LearnDataVault

http://learndatavault.com/

http://www.linkedin.com/dlinstedt


What IS a Data Vault? (Business

Definition)

• Data Vault Modelo Detail orientedo Historical traceabilityo Uniquely linked set of

normalized tableso Supports one or more

functional areas of business

ProcurementSales DeliveryContracts

FinancePlanning

Operations

Business KeysSpan / CrossLines of Business

Functional Area

• Data Vault Methodology– CMMI Level 5 Project Plan– Risk, Governance, Versioning– Peer Reviews, Release Cycles– Repeatable, Consistent,

Optimized– Complete with Best Practices

for BI/DW


What Does One Look Like?

Customer

Sat

Sat

Sat

F(x)

Customer

Product

Sat

Sat

Sat

F(x)

Product

Order

Sat

Sat

Sat

F(x)

Order

Elements:• Hub• Link• Satellite

Link

F(x)

Sat

Records a history of the interaction

Hub = List of Unique Business KeysLink = List of Relationships, AssociationsSatellites = Descriptive Data

Who’s Using It?


http://images.google.com/imgres?imgurl=http://iran-interlink.org/userfiles/Image/logos/USArmy.jpg&imgrefurl=http://www.iran-interlink.org/?mod=archive2008&usg=__nSN0gMyLixsdqTbHtOPjKnE5DNo=&h=300&w=300&sz=33&hl=en&start=13&tbnid=u3RlZ3vn0Sh13M:&tbnh=116&tbnw=116&prev=/images?q=US+Army&gbv=2&hl=en&safe=off

http://images.google.com/imgres?imgurl=http://www.pyramidsystems.com/images/Navy_Logo.jpg&imgrefurl=http://www.pyramidsystems.com/clients/clients_government.asp&usg=__tUODH1lYVE_qFe-pkoc1WLoH2eE=&h=900&w=900&sz=331&hl=en&start=16&tbnid=RXDagiaetLgM1M:&tbnh=146&tbnw=146&prev=/images?q=US+Navy&gbv=2&hl=en&safe=off

http://images.google.com/imgres?imgurl=http://www.forceforvictims.nl/SNSBank_logo.jpg&imgrefurl=http://apollo18.nl/2007/11/&usg=__EFd9uZB0Ydpj_53Kk-VxZ34f5QI=&h=137&w=422&sz=14&hl=en&start=5&tbnid=7p5PoSwCIH2sQM:&tbnh=41&tbnw=126&prev=/images?q=SNS+Bank+logo&gbv=2&hl=en&safe=off

http://images.google.com/imgres?imgurl=http://www.klaveradvies.nl/images/logo_belastingdienst.nl.gif&imgrefurl=http://www.klaveradvies.nl/belastingen.htm&usg=__v_xHzE3rf3zSuPiiIvVKWCHw_ho=&h=78&w=248&sz=9&hl=en&start=6&tbnid=1C94VSAO11haLM:&tbnh=35&tbnw=111&prev=/images?q=Belastingdienst+logo&gbv=2&hl=en&safe=off

http://images.google.com/imgres?imgurl=http://www.rcg.tv/html/eng/about/newsletter/images/logica_logo_3014.gif&imgrefurl=http://www.rcg.tv/html/eng/about/newsletter/2009jul.jsp&usg=__kJ5taG1txQZFxLZkAOQL9k-Aqlo=&h=228&w=400&sz=12&hl=en&start=1&tbnid=DmAr6vQ_7Bg8rM:&tbnh=71&tbnw=124&prev=/images?q=Logica+Logo&gbv=2&hl=en&safe=off

http://images.google.com/imgres?imgurl=http://www.ximep2008.org/newpage/microsoft-logo.jpg&imgrefurl=http://www.ximep2008.org/newpage/demo.html&usg=__t6JSpO_AYh-ne16BN-6hgYRu2z0=&h=686&w=2846&sz=80&hl=en&start=2&tbnid=qXPGv2e2QgNLeM:&tbnh=36&tbnw=150&prev=/images?q=Microsoft+Logo&gbv=2&hl=en&safe=off

http://images.google.com/imgres?imgurl=http://www.kringjubileum2008.nl/images/nutreco.jpg&imgrefurl=http://www.kringjubileum2008.nl/sponsoren.html&usg=__jTRrngkmZSORFpIgRW3eGTsOSog=&h=85&w=265&sz=8&hl=en&start=4&tbnid=5NkZLisWvM9rLM:&tbnh=36&tbnw=112&prev=/images?q=Nutreco+Logo&gbv=2&hl=en&safe=off

http://images.google.com/imgres?imgurl=http://www.travelatime.com/files/faa.gif&imgrefurl=http://www.travelatime.com/&usg=__OlPJZA4ROUGYKAekfygrTsUuWwk=&h=300&w=300&sz=49&hl=en&start=5&tbnid=OI1ffvea7Yx85M:&tbnh=116&tbnw=116&prev=/images?q=US+FAA&gbv=2&hl=en&safe=off

http://images.google.com/imgres?imgurl=http://www.telecompioneers.org/shared/content/sponsorcompanies/Qwest%20Logo%20(10%2008%2008).JPG&imgrefurl=http://www.telecompioneers.org/aboutus/sponsorcompanies/default.aspx&usg=__syj1dYYssVnzZOiI4Zf-Vea9n88=&h=655&w=1800&sz=114&hl=en&start=2&tbnid=mzzLj7oAsx1YrM:&tbnh=55&tbnw=150&prev=/images?q=Qwest+logo&gbv=2&hl=en&safe=off

http://images.google.com/imgres?imgurl=http://energyrogue.typepad.com/energyrogue/images/2008/06/14/diamler_tesla_logo.jpg&imgrefurl=http://energyrogue.typepad.com/energyrogue/2008/06/telsa-and-daiml.html&usg=___EmT_PQlugoPBwLvNultxhQqPAY=&h=531&w=500&sz=29&hl=en&start=5&tbnid=SXH1wK1Pg8uTNM:&tbnh=132&tbnw=124&prev=/images?q=Diamler+logo&gbv=2&hl=en&safe=off

http://www.tele2.com/index.html

http://www.tyson.com/

http://www.abnamro.com/index.cfm

http://www.google.com/imgres?imgurl=http://siteresources.worldbank.org/INTKAZAKHSTAN/Images/WB_logo.JPG&imgrefurl=http://www.worldbank.org.kz/WBSITE/EXTERNAL/COUNTRIES/ECAEXT/KAZAKHSTANEXTN/0,,contentMDK:22097087~menuPK:50003484~pagePK:2865066~piPK:2865079~theSitePK:361869,00.html&h=1000&w=983&sz=77&tbnid=b53FQKmYYkVgAM:&tbnh=149&tbnw=146&prev=/images?q=world+bank+logo&hl=en&usg=__vmfi8Wyz1Ag5-N8ptgcbWVzG6V0=&ei=uYzgSo2VJc7RjAfRyPGkBg&sa=X&oi=image_result&resnum=2&ct=image&ved=0CA0Q9QEwAQ

http://www.raytheon.com/

http://images.google.com/imgres?imgurl=http://www.summtech.com/logo/cigna_logo_wf.gif&imgrefurl=http://www.summtech.com/services/software_cigna.html&usg=__gNf9IBiiPoKzLwI4BB2wkorAQ48=&h=347&w=336&sz=9&hl=en&start=1&um=1&tbnid=ZB25EE8WIN2zxM:&tbnh=120&tbnw=116&prev=/images?q=cigna+health+care+logo&gbv=2&hl=en&safe=off&sa=N&um=1

http://images.google.com/imgres?imgurl=http://www.mayan-reservations.com/pictures/mellon-bank_thumb.gif&imgrefurl=http://www.mayan-reservations.com/mares-gav210-condo.html&usg=__guVS79TKD2t4BYqvjD4aiYC_yOM=&h=75&w=264&sz=5&hl=en&start=10&um=1&tbnid=8A6eNwCoxkSp1M:&tbnh=32&tbnw=112&prev=/images?q=mellon+bank&gbv=2&hl=en&safe=off&um=1

The PAIN!!Issues in Current EDW Projects



EDW Architecture: Generation 1

Sales

Finance

Contracts

Staging(EDW)

StarSchemas

Enterprise BI Solution

(batch)

Staging + History

Complex Business

Rules

Complex Business Rules+Dependencies

Conformed DimensionsJunk Tables

Helper TablesFactless Facts


Kick-Starting Data Warehousing

HR Asks IT to build the FIRST Data Warehouse / Prototype

1.

IT Says… OK: $125k and 90

days…

2.

HR Says:Great! Get

Started

3.


Everyone’s Happy!

IT Delivers. On-Time & In Budget!

Customer_IDCustomer_NameCustomer_AddrCustomer_Addr1Customer_CityCustomer_StateCustomer_ZipCustomer_PhoneCustomer_TagCustomer_ScoreCustomer_RegionCustomer_StatsCustomer_PhoneCustomer_Type

First Star!

Customer_IDCustomer_NameCustomer_AddrCustomer_Addr1Customer_CityCustomer_StateCustomer_ZipCustomer_PhoneFact_ABCFact_DEFFact_PDQFact_MYFACT




4.

HR Says:Thank-you! We’re

Happy!

5.

So Where’s the PAIN?



The PAIN is RIGHT HERE!!

Contracts Sees Success, wants the same for their systems.

1.

IT Says… Ok, but… It won’t be $125k and 90

days…Because we have to “merge

it” with HR” it will be $250 and 180 days.

2.

Contracts Says:Ouch! That’s not

reasonable, but we need it, so go ahead…

3.


And HERE….

Finance, Sales, and Marketing want in….IT Says… Ok, but… It won’t be $250k and 90 days… Because we

have to “merge it” with HR and Contracts it will be $350k and 250 days.

And this continues….Business Says...“Can’t you just make-a-copy of the Star

Schema, and give me my own for cheaper & less time?


Silo Building / IT Non-Agility

We built our own because IT costs too much

SALES

We built our own because IT took too long

FINANCE

We built our own because we need customized dimension data

MARKETING

First Star

Why is this happening? What’s Causing this Problem?

Root Cause of Pain: Re-Engineering!


IT is forced to Re-Engineer ETL loading code + SQL BI Queries WHENEVER:• WHENEVER table

structures change• New systems are introduced

• Business Rules Change • (causing ETL Loading to change, and forcing Engineers to RELOAD existing data)

1. Adding fields to Dimensions

2. Adding fields to Facts


Customer_IDCustomer_NameCustomer_AddrCustomer_Addr1Customer_CityCustomer_StateCustomer_ZipCustomer_PhoneFact_ABCFact_DEFFact_PDQFact_MYFACT





3. Adding Dimensions to Facts

Why Re-Engineering?


Require Re-Engineering!

Adding fields to a conformed dimension….

Adding fields to a shared fact….

Require adding/changingFields in target tables!

Changing code to match new business rules…

Other Pains?


Dimension-Itis?

Deformed Dimensions?

IT – Non-Agility?

WHAT ABOUT THE “DATA” YOU DON’T SEE?

WHAT ABOUT THE “BAD” DATA LEFT IN THE SOURCE SYSTEMS?

The SolutionGo the Data Vault Route!



EDW Architecture: Generation 2

Sales

Finance

Contracts

StagingDV

EDW

StarSchemas

ErrorMarts

ReportCollections

Enterprise BI SolutionSOA

(real-time)

(batch)

(batch)

Business Rules Downstream!(the Lens Filter)


Unstructured Data And Data Vault

Unstructured Data Sets Ontologies/Taxonomies

• Email• Docs• Images• Movies• Sound

Unstructured Processing Engine

Data Vault EDW

Joins through LINK Structures

On-DemandCubes


IT Agility

Source StagingData Vault

(EDW)

RAW“what-is”

StarSchemas

1. Fast Load & Fast Integration

ComplexBusiness

Rules

BusinessDriven

StarSchemas

3. IT Implementation of Business Rules

ETL-T

2. Business Gap Analysis•Unknown Time…•Business Requirements•Start new phase

What are the Facts Jack?


Generation 1 EDW’s tried to provide“One version of the truth”

Generation 2 (Data Vaults) provide…“One version of the facts, for each point in time.”


Business Gap Analysis

GapAnalysis

The Way Business Perceives it’s business to be running

The way the source systems see the business running.

GapAnalysis

OperationalReports

DynamicCubes(Data Marts)


Secured/Protected Information Systems

Non-Classified DV

HubHub

SatSatSatSat

HubHub

SatSat

LinkLinkSatSat

Classified Data Vault

HubHub

SatSatSatSat

HubHub

SatSatSatSat

LinkLink

SatSat

Hub

SatSat

Link

Sat

Sat

• Model changes are absorbed seamlessly into the classified system• Classified world can add all their own structures while maintaining congruence with standard

unclassified Data Vault

Data Copy

Model Copy

Yellow = New Tables


Extensibility Factor

ProductSupplierLink

Product ShippedDates

BilledAmounts Product

Quantities

Suppliers

Descriptions

Stock Quantities

Address

Rating Score

Products

Descriptions

Stock Quantities

Availability Dates

Defect Reasons

Existing EDWNo Impact!

New AdditionsNew Code

Where’s the Solution?


Handle Changes Wherever… Whenever… with EASE!

Re-Engineering

The Three vehicles…Pros and Cons of the Modeling Methodologies



3rd Normal Form Pros/Cons as an

EDWPROS (as 3NF)• Many to many linkages• Handle lots of information• Tightly integrated information• Highly structured• Conducive to near-real time loads• Relatively easy to extend

CONS (as EDW)• Time driven PK issues• Parent-child complexities• Cascading change impacts• Difficult to load• Not conducive to BI tools• Not conducive to drill-down• Difficult to architect for an

enterprise• Not conducive to spiral/scope

controlled implementation• Physical design usually doesn’t

follow business processes

http://www.hankstruckpictures.com/pix/trucks/loeb/mack_r7_3_1984.jpg


Star Schema Pros/Cons as an EDW

PROS (as Data Mart)• Good for multi-dimensional analysis• Subject oriented answers• Excellent for aggregation points• Rapid development / deployment• Great for some historical storage

CONS (as EDW)• Not cross-business functional• Use of junk / helper tables• Trouble with VLDW• Unable to provide integrated

enterprise information• Can’t handle ODS or exploration

warehouse requirements• Trouble with data explosion in

near-real-time environments• Trouble with updates to type 2

dimension primary keys• Trouble with late arriving data in

dimensions to support real-time arriving transactions

• Not granular enough information to support real-time data integration


Data Vault Pros/Cons as an EDW

PROS (as EDW)• Supports near-real time and

batch feeds• Supports functional business

linking• Extensible / flexible• Provides rapid build / delivery

of star schema’s• Supports VLDB / VLDW• Designed for EDW• Supports data mining and AI• Provides granular detail• Incrementally built

CONS (as EDW)• Not conducive to OLAP

processing• Requires business

analysis to be firm• Introduces many join

operations

http://www.autotrader.com/findacar/images.jtmpl?car_id=151684771&dealer_id=104722&cardist=9&&make=CAD&address=80123&search_type=used&model=ESCALAESV&max_price=&min_price=&start_year=1984&end_year=2005&car_year=2003&distance=25&advcd_on=n&advanced=n&color=&


The Three Vehicles…

• Which would you use to win a race?• Which would you use to move a house?• Would you adapt the truck and enter a race with Porches and expect to

win?

#1 complaint about DV architecture

So you want to deal with Joins do you?


Joins, Everywhere!


Yes, the DV is full of joins but…These are highly normalized tables (thin & Narrow), reducing I/O’s to read large numbers of rows, at high speed, in parallel. Joins occur in RAM instead of on disk. The Optimizer is given a chance to “drop tables” from the join that aren’t necessary.

When Parallelism is too much…• Not enough CPU or RAM to handle the extra work-load• Not enough rows being queried, (the overhead of starting the threads

takes longer than an original scan.

End Result? The DV Scales to the Petabyte Levels when necessary…

Mathematics Behind the Data Vault Model

*** The Data Vault is BACKED by Mathematical Principles***

• Parallel versus sequential execution models• Set Logic• I/O Bandwidth & Throughput• Compression (for query performance gains)• Process Repeatability (tuning & predictability

measurements)• RAM versus electromagnetic disk (Solid-State

Drives are not measured)

• http://osl.cs.uiuc.edu/docs/IPDPS-TR04/TCA_TR04.pdf


http://osl.cs.uiuc.edu/docs/IPDPS-TR04/TCA_TR04.pdf

http://osl.cs.uiuc.edu/docs/IPDPS-TR04/TCA_TR04.pdf

Know when to hold ‘em, know when to fold ‘em

When to use DV, and when not…


The Challenger….


The challenger says:• My system works fine, why should I use the Data Vault? • I don’t have volume problems…• I don’t have compliance/auditability problems…• I don’t have real-time problems…• My system produces matching results across lines of business…• I’ve never had to “re-state” the data in the warehouse…• I can still build new marts, and conform dimensions in 30 days or less…• My business doesn’t acquire new systems often (if ever)• My incoming data sets don’t change

I Say…That’s wonderful, don’t fix what’s broken. Have a nice day, oh- but call me when or if you ever run into these problems…

When to Apply the Data Vault


• Scalability• Auditability• Flexibility• Adaptability

Benefits:

• IT Agility• IT and Business Accountability• Reduction in Spread-Marts• Corporate Asset Development• Money Savings• Risk Mitigation• Successful EDW Implementations

Leads To…

How to build a data vault

In 10 easy steps…


Step 1


Identify your business processes, followed by your business keys (that are used to identify the data that flows through the business processes)

** NOTE: Along the way, document your assumptions, document your reasons for choosing keys, and modeling designs, develop a list of questions to be answered by business users…

Step 2


Identify the issues/problems that might be carried with the identified business keys, annotate the risks, and mitigate each one.

Step 3


Identify the units of work, the associations – LINK tables, where keys combine to form a notion, a concept, and a relationship.

Step 4


Identify the descriptive data that belongs to SINGLE Hub Keys, ensure that the data doesn’t represent or rely on a relationship.

Step 5


Identify the Satellite data that depends on relationships – move it to the appropriate LINK table.

HINT: If you “want” to put a Foreign Key in a Satellite, you have a clear sign that the Satellite is in the WRONG place, and needs to be assigned to a LINK table rather than a HUB.

Step 6


Scope the Model Down to a managable chunk. Implement the first two Hubs, Hub Satellites, and first Link. BUILD IN INCREMENTS!

Step 7


Setup the key generation load routines, setup the staging area, and begin loading data.

Step 8


Review any “truncation” errors, or any data-type conversion problems, fix the staging area, and remove duplicates.

Step 9


Begin Loading the Data Vault. Load all Hubs, then all Hub Satellites, Then all Links, and finish with All Link Satellites.

Step 10


Reconcile the Data Vault to the source system, then build a first data mart from the results. Bring business value FAST!

Instructor led lab



10 minutes to find the Hubs….


Possible Hubs From Northwind


10 Minutes to find the Links…


Possible Links From Northwind


10 minutes to find the Satellites…


Possible Satellites From Northwind

What did we learn?• We often deal with more than 1 system at a

time… this was a lab with only one model.• We didn’t have any business requirements that

we might need to answer questions, but doesn’t that reflect real-life?

• The data set is extremely dirty (you never have that in your systems right?)

• Time Zone based data can be a problem• Lack of metadata causes integration issues and

modeling decisions


57

The Experts Say…“The Data Vault is the optimal choice for modeling the EDW in the DW 2.0 framework.” Bill Inmon

“The Data Vault is foundationally strong and exceptionally scalable architecture.” Stephen Brobst

“The Data Vault is a technique which some industry experts have predicted may spark a revolution as the next big thing in data modeling for enterprise warehousing....” Doug Laney

58

More Notables…

“This enables organizations to take control of their data warehousing destiny, supporting better and more relevant data warehouses in less time than before.” Howard Dresner

“[The Data Vault] captures a practical body of knowledge for data warehouse development which both agile and traditional practitioners

will benefit from..” Scott Ambler

59

Where To Learn More• The Technical Modeling Book:

http://LearnDataVault.com

• The Discussion Forums: & eventshttp://LinkedIn.com – Data Vault Discussions

• Contact me:http://DanLinstedt.com - web [email protected] - email

• World wide User Group (Free)http://dvusergroup.com

http://learndatavault.com/

http://linkedin.com/

http://danlinstedt.com/

mailto:[email protected]

http://dvusergroup.com/

irm uk - 2009: dv modeling and methodology

Technology

descriptive data

data warehouse

com4 data vault methodology

dimensions business

root cause of pain

time photographer25

list of relationships

edw architecture