the data warehouse etl toolkit - chapter 06

8/9/2019 The Data Warehouse ETL Toolkit - Chapter 06

1/77

The Data WareHouse ETL

ToolkitVSV Training

Chapter 6: Delivering Fact TablesPrepare b!: Tho H"#$% & Hien

$%'(E$Date: )*+),+,))-


2/77

6. Delivering Fact Tables

/. The 0asic Structure o1 a Fact Table

,. %uaranteeing 2e1erential 3ntegrit!

4. Surrogate 5e! Pipeline/. 'sing the Diension 3nstea o1 a Lookup Table

7. Funaental %rains/. Transaction %rain Fact Tables

,. Perioic Snapshot Fact Tables

4. #ccuulating Snapshot Fact Tables

8. Preparing 1or Loaing Fact Tables

/. 9anaging 3nees,. 9anaging Partitions

4. "ut;itting the 2ollback Log

7. Loaing the Data


3/77

5. Preparing for Loading Fact Tables (cont.)

8. 3ncreental Loaing

6. 3nserting Facts


4/77

6. Factless Fact Tables


5/77

13.Delivering Dimensional Data toOLAP !bes

/. Cube Data Sources

,. Processing Diensions

4. Changes in Diension Data

7. Processing Facts

8. 3ntegrating "L#P Processing into the ETLS!ste

6. "L#PWrapup

6. Delivering Fact Tables=cont.>


6/77

6.) 3ntrouction

Fact tables hol the easureentso1 an enterprise.

Fact tables contain easureentsBan iension tables contain thecontet surrouning easureents.


7/77

6./. The 0asic Structure o1 aFact Table

Ever! 1act table is e?ne b! the grainof the table =The grain o1 the 1act tableis the e?nition o1 the easureent

event>

#ll 1act tables possess a set o1 1oreign

ke!s connecte to the iensions thatprovie the contet o1 the 1act tableeasureents


8/77

6./ =cont.>


9/77

6.,. %uaranteeing 2e1erential3ntegrit!

There are onl! t;o ;a!s to violatere1erential integrit! in a iensionalschea:

/. Loa a 1act recor ;ith one or oreba 1oreign ke!s.

,. Delete a iension recor ;hosepriar! ke! is being use in the 1acttable.


10/77

6., =cont.>Three ain places in the ETL pipeline ;here

re1erential integrit! can be en1orce. The! are:

/.Care1ul bookkeeping an ata preparation ustbefore loading the fact table recors into the ?naltablesB couple ;ith care1ul bookkeeping be1ore

eleting an! iension recors

,. En1orceent o1 re1erential integrit! in theatabase itsel1 at the moment o1 ever! 1act tableinsertion an ever! iension table eletion

4. Discover! an correction o1 re1erential integrit!violations after loading has occurred by regularlyscanning the fact table, looking for ba 1oreignke!s


11/77

6., =cont.>


12/77

The @ueries checking re1erentialintegrit! ust be o1 the 1or:

selectf.product_key

fromfact_table f

"#eref.product_key not in(selectp.product_key fromproduct_dimension p)

6., =cont.>


13/77

6.4. Surrogate 5e! Pipeline

#ll recors to be loae into the 1acttable are current

When builing a 1act tableB the ?nalETL step is converting the naturalke!s in the ne; input recors intothe correctB conteporar! surrogateke!s.


14/77

6.4 =cont.>


15/77

6.4./. 'sing the Diension 3nstea o1

a Lookup Table The lookup table approach ;orks best;hen the over;heling 1raction o1 1actrecors processe each a! are

contemporary

# signi?cant nuber o1 1act recors arelate arriving ;hat happene

The iension ust be the source 1or thecorrect surrogate ke!


16/77

6.7. Funaental %rains

Ever! 1act table shoul have oneBand only one, grain.

The three kins o1 1act tables are: thetransaction grainB the perioicsnapshotB an the accuulatingsnapshot.


17/77

6.7./. Transaction $rain FactTables The transaction grain represents an

instantaneous easureent at aspeci?c point in space an tie

The tin! atoic easureentst!pical o1 transaction grain 1act tableshave a large nuber o1 iensions

3n environents like a retail storeBthere a! be onl! one transactiont!pe =the retail sale> being easure.


18/77

6.7./. =cont.>


19/77

6.7./. =cont.>


20/77

6.7.,. Perioic Snapshot FactTables

The perioic snapshot represents aspan o1 tieB regularl! repeate.

The ate iension in the perioicsnapshot 1act table re1ers to theperio.

Perioic snapshot 1act tables havecopletel! preictable sparseness


21/77

Perioic snapshot 1act tables havesiilar loaing characteristics tothose o1 the transaction grain tables

Ho;everB there are t;o soe;hatiGerent strategies 1or aintainingperioic snapshot 1act tables.

6.7.,. Perioic Snapshot FactTables =cont.>


22/77

6.7.4. #ccuulating Snapshot FactTables The accuulating snapshot 1act table is

use to escribe processes that have ae?nite beginning an enB such as orer1ul?llentB clais processingB an ost;orko;s

The grain o1 an accuulating snapshot 1acttable is the coplete histor! o1 an entit!1ro its creation to the present oent

#ccuulating snapshot 1act tables haveseveral unusual characteristics


23/77

6.7.4. =cont.>


24/77

6.8. Preparing 1or Loaing FactTables

Ho; to buil eIcient loa processesan overcoe coon obstacles


25/77

6.8./. 9anaging 3nees

3nees are per1orance enhancersat @uer! tieB but the! areper1orance killers at loa tie.


26/77

3n a nutshellB per1or the steps that 1ollo;to prevent table inees 1ro causing abottleneck in !our ETL process:

/. Segregate upates 1ro inserts.,. Drop an! inees not re@uire to support

upates.

4. Loa upates.

7. Drop all reaining inees.

8. Loa inserts =through bulk loaer>.

6. 2ebuil the inees.

6.8./. 9anaging 3nees=cont.>


27/77

6.8.,. 9anaging Partitions

Partitions allo; a table =an itsinees> to be ph!sicall! ivie intominitables 1or ainistrative purposesan to iprove @uer! per1orance.

The partitions o1 a table are under thecovers, hidden from the users.

"nl! the D0# an ETL tea shoul bea;are o1 partitions.


28/77

The ost coon partitioning strateg! on1act tables is to partition the table b! theate ke!.

Tables that are partitione b! a ateinterval are usuall! partitione b! !earB@uarterB or onth.

'nless !our D0# tea takes a proactiverole in ainistering !our partitionsB theETL process ust anage the.

6.8.,. 9anaging Partitions=cont.>


29/77

Suppose !our 1act table is partitione b! !ear an

the ?rst three !ears are create b! the D0#tea. When !ou attept to loa an! ata a1terDeceberB 4/B ,))7B in "racle !ou receive the1ollo;ing error:

ORA-!!""# inserted partition key is beyond highestlegal partition key

#t this pointB the ETL process has a choice:

$oti1! the D0# teaB ;ait 1or the to anuall!create the net partitionB an resue loaing.

D!naicall! a the net partition re@uire tocoplete loaing.



30/77

selectma%=ateJke!> fromKST#%EJF#CTJT#0LE

copare ;ith

selecthighJvalue fromallJtabJpartitions

"#eretableJnae M KF#CTJT#0LEK

andpartitionJposition M =selectma%=partitionJposition>

fromallJtabJpartitions "#eretableJnae M KF#CTJT#0LEK>



31/77

6.8.4. "ut;itting the 2ollbackLog2easons ;h! the ata ;arehouse oesnot nee rollback logging inclue:

#ll ata is entere b! a anageprocessNthe ETL s!ste.

Data is loae in bulk.

Data can easil! be reloae i1 a loaprocess 1ails.


32/77

6.8.7. Loaing the Data

The initial loa o1 a ne; table has auni@ue set o1 challenges. The priar!challenge is hanling the onetieiense volue o1 ata.

Separate inserts 1ro upates

'tilie a bulkloa utilit!

Loa in parallel

9iniie ph!sical upates

0uil aggregates outsie o1 the atabase


33/77

6.8.8. 3ncreental Loaing

The increental loa is the processthat occurs perioicall! to keep theata ;arehouse s!nchronie ;ith itsrespective source s!stes.

ETL routines that loa ataincreentall! are usuall! a result o1the process that initiall! loae thehistoric ata into the ata ;arehouse.


34/77

6.8.6. 3nserting Facts

Fact tables are too iense toprocess via SOL &'*Tstateents.

6 8 < 'pating an Correcting


35/77

6.8.


36/77

6.8.-. $egating Facts

The negative easures in the reversing1act table recor cancel out the originalrecor.

9an! reasons eist 1or negating anerror rather than taking otherapproaches to correcting 1act ata.

"ther reasons 1or negating 1actsBinstea o1 upating or eletingB involveata volue an ETL per1orance.


37/77

6.8.*. 'pating Facts

'pating ata in 1act tables can be aprocessintensive eneavor.

The best approach to upating 1actata is to *F*+ the table via thebulkloa utilit!.


38/77

6.8./). Deleting Facts

Deleting 1acts 1ro a 1act table is1orbien in ata ;arehousing.

31 !our business re@uires eletionsBt;o ;a!s to hanle the eist: Ph!sical eletes

Logical eletes


39/77

6.8.//. Ph!sicall! DeletingFactsPh!sicall! eleting 1acts eans ata is

peranentl! reove 1ro the ata;arehouse.

'ever,users to think in ters o1toa!sre@uireents because it isbase on their current ;a! o1

thinking about the ata the! use ee, users have no iea ;hat eists

in ra; ata


40/77


41/77

6.8./,. Logicall- DeletingFacts# logical elete entails the utiliation

o1 an aitional colun naedeleted.


42/77

6.6. Factless Fact Tables

The grain o1 ever! 1act table is aeasureent event.


43/77

#nother coon t!pe o1 1actless 1acttable represents a coverage

6.6. Factless Fact Tables =cont.>


44/77

6.


45/77

Three approaches can be use in this case:

/. 9aintain a 1ull T!pe , iension oG to thesie

,. 0uil the priar! iension as a 1ull T!pe, iension.

4. 0uil the priar! iension as a 1ull T!pe, iension an siultaneousl! ebethe natural ke! o1 the iension in the1act table alongsie the surrogate ke!

6.


46/77

6.-. %race1ul 9oi?cations

There are 1our t!pes o1 grace1uloi?cations to iensional scheas:

/. #ing a 1act to an eisting 1act table at thesae grain

,. #ing a iension to an eisting 1act tableat the sae grain

4. #ing an attribute to an eisting iension7. 3ncreasing the granularit! o1 eisting 1actan iension tables.


47/77

6.-. %race1ul 9oi?cations=cont.>


48/77

The ?rst three t!pes raise the issueo1ho;to populate the ol histor! o1the tables prior to the aition o1 the

1actB iensionB or attribute.



49/77

When the change is vali onl! 1rotoa! 1or;arB ;e hanle the ?rstthree oi?cations as 1ollo;s:

/.Adding a $act

,.Adding a %imension

4.Adding a %imension Attribute


6 * 9 lti l ' it 1 9


50/77

6.*. 9ultiple 'nits o1 9easurein a Fact Table


51/77


52/77

6.//. Late #rriving Facts

31 ;e have been tiestaping the iension recors inour T!pe , SCDsB our processing involves the 1ollo;ingsteps:

/.For each iensionB ?n the corresponing iensionrecor in eGect at the tie o1 the purchase.

,. 'sing the surrogate ke!s 1oun in the each o1 theiension recors 1ro Step /R replace the naturalke!s o1 the latearriving 1act recor ;ith the surrogateke!s.

4. 3nsert the latearriving 1act recor into the correctph!sical partition o1 the atabase containing the other1act recors 1ro the tie o1 the latearrivingpurchase.


53/77

6./,. #ggregations

The single ost raatic ;a! toaGect per1orance in a large ata;arehouse is to provie a proper seto1 aggregate =suar!> recors that

coeist ;ith the priar! base recors.

#ggregate navigation is a stanar

ata ;arehouse topic that has beeniscusse etensivel! in literature

i


54/77

3n a properl! esigne ata ;arehouseenvironentB ultiple sets o1 aggregates arebuiltB representing coon grouping levels;ithin the ke! iensions o1 the ata ;arehouse.

#ggregate navigation has been e?ne ansupporte onl! 1or iensional ata ;arehouses.There is no coherent approach 1or aggregatenavigation in a noralie environent.

#n aggregate navigator is a piece o1 ile;arethat sits bet;een the re@uesting client an theD09S.

6./,. #ggregations =cont.>

i = >


55/77

#n aggregate navigator intercepts theclients SOL anB ;herever possibleBtrans1ors baselevel SOL into aggregatea;are SOL.

The aggregate navigator unerstans ho;to trans1or baselevel SOL into aggregatea;are SOL because the navigator usesspecial etaata that escribes the ata;arehouse aggregate port1olio.


6 /, # i = >


56/77


6 /, # i = >


57/77

# goo aggregate progra 1or a largeata ;arehouse shoul:

/. Provie raatic per1orance gains 1or asan! categories o1 user @ueries as possible

,. # onl! a reasonable aount o1 etraata storage to the ;arehouse.

4. 0e copletel! transparent to en usersan to application esignersB ecept 1or theobvious per1orance bene?ts.


6 /, # ti = t >


58/77

# goo aggregate progra 1or a largeata ;arehouse shoul:

7. #Gect the cost o1 the ataetracts!ste as little as possible.

8. #Gect the D0#s ainistrativeresponsibilities as little as possible.


6 /, # i = >


59/77



60/77

6./,./. Design 2e@uireent A/ #ggregates ust be store in their

o;n 1act tablesB separate 1ro baselevel ata. Each istinct aggregationlevel ust occup! its o;n uni@ue 1act

table.

The separation o1 aggregates into their

o;n 1act tables is ver! iportant anhas a ;hole series o1 bene?cial sieeGects.

6 /, , D i 2 i t A,


61/77

The iension tables attache to theaggregate 1act tables ustB ;hereverpossibleB be shrunken versions o1 the

iension tables associate ;ith thebase 1act table.

6./,.,. Design 2e@uireent A,

6 /, , D i 2 i t A,


62/77

6./,.,. Design 2e@uireent A,=cont.>

6 /, 4 D i 2 i t A4


63/77

The base 1act table an all its relateaggregate 1act tables can be associatetogether as a family of schemas so thatthe aggregate navigator kno&s &hich

tables are relate to each other.

The registration o1 this 1ail! o1 1acttablesB together ;ith the associate

1ullsie an shrunken iensiontablesB is the sole etaata neee inthis esign.

6./,.4. Design 2e@uireent A4

6 /, 7 D i 2 i t A7


64/77

Force all SOL create b! an! en useror application to re1er eclusivel! tothe base 1act table an its associate1ullsie iension tables.

This esign re@uireent pervaes alluser inter1aces an all en user

applications.

6./,.7. Design 2e@uireent A7

. . . #inistering #ggregationsB


65/77

3ncluing9aterialie Vie;s

There are a nuber o1 iGerentph!sical variations o1 aggregationsBepening on the D09S an the

1ronten tools. There are t;o 1unaental

approaches to aggregating

navigation at the tie o1 this ;riting.

. . . n s er ng ggrega onsB3ncluing


66/77

%enerall!B a given aggregate 1act tableshoul be at least tenties saller thanthe base 1act table in orer to ake thetraeoG bet;een ainistrative overhea

an per1orance gain ;orth;hile Large har;arebaseB parallelprocessing

architectures gain eactl! the saeper1orance avantages 1ro aggregatesas uniprocessor s!stes ;ith conventionalisk storageB since the gain coes sipl!1ro reucing total 3+".

3ncluing9aterialie Vie;s =cont.>

6 /4 Delivering Diensional


67/77

6./4. Delivering DiensionalData to "L#P Cubes

"L#P servers eliver t;o priar!1unctions

Ouer! per1orance

#nal!tic richness

The best source 1or an "L#P cube isa iensional ata ;arehouse storein an 2D09S.

6 /4 / Cube Data Sources


68/77

6./4./. Cube Data Sources The various serverbase "L#P proucts

an versions have iGerent 1eatures.

31 !ou are sourcing !our cube 1ro at

?lesB one o1 the last steps in !our ETLprocess isB obviousl!B to ;rite out theappropriate atasets.

#nal!e an! @ueries that !our cubeprocessing issues against the relationalata ;arehouse.

6 /4 , Processing Diensions


69/77

6./4.,. Processing Diensions

ust as relational iensions areprocesse be1ore the 1act tables thatuse theB so ust "L#P iensionsbe processe be1ore 1acts

(our ETL s!ste esign nees to bea;are o1 a 1e; characteristics o1

"L#P iensions.


70/77

"L#Pservers hanle iGerent kins o1iensionata changes copletel! iGerentl!

Depening on the costs an s!ste usageB !ou

coul ecie to esign !our s!ste to: Let the "L#P an relational ata ;arehouse

atabases iverge

5eep the "L#P cubes in s!nch ;ith the relationalata ;arehouse b! halting relational processing as

;ell 5eep the "L#P cubes in s!nch ;ith the relational

ata ;arehouse b! accepting the epensivereprocessing operation uring a nightl! loa

6./4.4. Changes in Diension

Data

6 /4 7 Processing Facts


71/77

6./4.7. Processing Facts

Serverbase "L#P proucts are capableo1 anaging ver! large volues o1 ataan are increasingl! use to hol ata atthe sae grain as the relational ata;arehouse.

9ost serverbase "L#P proucts supportsoe 1or o1 increental processingRothers support onl! 1ull processing.

Loaing into a partition is an appealing;a! to loa a subset o1 the cubes ata

6 /4 7 Processing Facts =cont >


72/77

Partitioning the "L#P cube canprovie signi?cant bene?ts 1or both@uer! an processing per1orance.

Soe "L#P servers also support trueincreental 1act processing.

6./4.7. Processing Facts =cont.>

6 /4 8 3ntegrating "L#P Processing


73/77

6./4.8. 3ntegrating "L#P Processinginto the ETL S!ste

Technologies that inclue ETL an "L#PoGerings 1ro a single venor provieore elegant integration.

9an! s!stes can 1ull! process the entire"L#P atabase on a regular scheuleBusuall! ;eekl! or onthl!.

For larger s!stesB a coon integration

structure inclues aing the "L#Piension processing as the ?nal step ineach iension table processing branchBouleB or package in !our ETL s!ste.

6 /4 6 "L#P Wrap up


74/77

6./4.6. "L#P Wrapup

31 an "L#P atabase is part o1 !ourata ;arehouse s!steB it shoul beanage rigorousl!


75/77

Suar!

We e?ne the 1act table as thevessel that hols all nuericeasureents o1 the enterprise.

We sa; that re1erential integrit! ishugel! iportant to the proper

1unctioning o1 a iensional scheaBan ;e propose three places ;herere1erential integrit! can be en1orce.


76/77

We sho;e ho; to buil a surrogateke! pipeline 1or ata ;arehouses thataccuratel! track the historical

changes in their iensional entities We escribe the structure o1 the

three kins o1 1act tables: transaction

grainB perioic snapshot grainB anaccuulating snapshot grain

Suar! =cont.>


77/77

We then propose a nuber o1 speci?ctechni@ues 1or hanling grace1uloi?cations to 1act an iension

tablesB ultiple units o1 easureentBlatearriving 1act ataB an builingaggregations.

# specialie section on loaing "L#Pcubes.

Suar! =cont.>

the data warehouse etl toolkit - chapter 06

Documents