sunita sarawagi iit bombay sunita analyzing large multidimensional databases
TRANSCRIPT
Sunita SarawagiIIT Bombay
http://www.it.iitb.ac.in/~sunita
Analyzing large multidimensional databases
Data Analysis in decision support systems
Data analysis understanding the effect of various factors on a target variable Factors: region, time, sales channel Target: profit, sales volume
Tools for data analysis: SQL queries/reports: slow, manual, painful Multidimensional tools: OLAP, very popular Statistical packages: sophisticated Data mining: automated, lot of interest and hype
but several hurdles to meaningful adoption
OLAP (On line Analytical Processing)
Data viewed as a multidimensional cube where factors are dimensions with hierarchies targets are values within cells
Analysis happens through fast interactive browsing of aggregates
Market share in 2002: US $3.5 billion Vendors: Microsoft, Hyperion, Cognos,
Business Objects about 30 such vendors
Multidimensional Data analysis
Sales volume as a function of product, month, and region
Pro
duct
Regio
n
Month
Dimensions: Product, Location, TimeHierarchical summarization paths
Industry Region Year
Category Country Quarter
Product City Month Week
Office Day
Typical OLAP Operations Roll up (drill-up): summarize data
by climbing up hierarchy or by dimension reduction
Drill down (roll down): reverse of roll-up from higher level summary to lower level summary
or detailed data, or introducing new dimensions Slice and dice:
project and select Pivot (rotate):
reorient the cube, visualization, 3D to series of 2D planes.
Limitation of OLAP-based analysis
OLAP products provide a minimal set of tools for analysis: simple aggregates selects/drill-downs/roll-ups on the
multidimensional structure
Heavy reliance on manual operations for analysis tedious on large data with multiple dimensions and
levels of hierarchy
I3: Intelligent, Interactive Investigation of multidimensional data
lightweight automation of tedious multi-step tasks
Three examples: Diff for specific why questions at aggregate level
most compactly represent the answer that user can quickly assimilate
Generalize from detailed data to more general cases: expand scope of problem case as far out as possible
Inform of interesting regions in data
The Diff operator
Unravel aggregate data
Total sales dropped 30%in Europe. Why?
What is the most compact answer that user can quickly assimilate?
Solution
A new DIFF-operator added to OLAP systems that provides the answer in a single-step is easy-to-assimilate and compact --- configurable by user.
Obviates use of the lengthy and manual search for reasons in large multidimensional data.
Example query
Plat_User (All)Plat_Type (All)Platform (All)Prod_Group (All)Prod_Category (All)Product (All)
Sum of Revenue YearGeography 1990 1991 1992 1993 1994Asia/Pacific 1440 1947 3454 5576 6310Rest of World 2170 2154 4577 5204 5510United States 6545 7524 10947 13545 15817Western Europe 4552 6061 10053 12578 13501
Plat_User (All)Plat_Type (All)Platform (All)Prod_Group (All)Prod_Category (All)Product (All)
Sum of Revenue YearGeography 1990 1991 1992 1993 1994Asia/Pacific 1440 1947 3454 5576 6310Rest of World 2170 2154 4577 5204 5510United States 6545 7524 10947 13545 15817Western Europe 4552 6061 10053 12578 13501
Compact answerPRODUCT PLAT_USERPLAT_TYPEPLATFORM 1990 1991 RATIO ERROR(All)- (All)- (All) (All) 1620 1820 1.1 34Operating SystemsMulti (All)- (All) 254 198 0.8 23Operating SystemsMulti Other M. Multiuser Mainframe IBM98 2 0.0 0Operating SystemsSingle Wn16 (All) 94 11 0.1 0*Middleware & Oth.UtilitiesMulti Other M. Multiuser Mainframe IBM101 10 0.1 0EDA Multi Unix M. (All) 0.4 76 211.7 0EDA Single Unix S. (All) 0.1 13 210.8 0
Example: explaining increasesPlat_Type (All)Geography (All)Prod_Group Soln
Sum of Revenue YearProd_Category 1990 1991 1992 1993 1994Cross Ind. Apps 1975 2484 4564 7407 8150Home software 294 575Other Apps 843 1172 3436Vertical Apps 898 1461 2827 7947 8663
Compact answerPRODUCT GEOGRAPHYPLAT_TYPEPLATFORM 1992 1993 RATIO ERROR(All)- (All)- (All)- (All) 2113 2763 1.3 200Manufacturing - Process(All) (All) (All) 26 702 27.1 250Other Vertical Apps(All)- (All)- (All) 20 1858 91.4 251Other Vertical AppsUnited StatesUnix S. (All) 8 77 9.6 0Other Vertical AppsWestern EuropeUnix S. (All) 7 96 13.2 0Manufacturing - Discrete(All) (All) (All) 1135 0Health Care (All)- (All)- (All)- 7 820 118.2 98Banking/FinanceUnited StatesOther M. (All) 341 239 0.7 60Mechanical CADUnited States(All) (All) 328 243 0.7 34
Model for summarization
The two aggregated values correspond to two subcubes in detailed data.
Products (All)Geography (All)
Year 1990 1991 19922000 1800
Year 1990Products
Geography OS DBMS Prog
Asia 100 80 80
USA 100 200 400
UK 140 100 56
Year 1991Products
Geography OS DBMS Prog
Asia 80 90 70
USA 120 240 480
UK 140 60 56
Cube-A Cube-B
Detailed answersPRODUCT GEOGRAPHY PLATFORM Y1992 Y1993 RATIOOther Vertical AppsWestern Europe Multiuser Minicomputer OpenVMS 99.9 Other Vertical AppsAsia/Pacific Single-user MAC OS 92.5 Other Vertical AppsRest of World Multiuser Mainframe IBM 88.1 Other Vertical AppsWestern Europe Single-user UNIX 7.3 96.3 13.2Other Vertical AppsUnited States Multiuser Minicomputer Other 97.2 Other Vertical AppsUnited States Multiuser Minicomputer OS/400 99.5 Other Vertical AppsAsia/Pacific Multiuser Minicomputer OS/400 99.6 EDA Western Europe Multiuser UNIX 192.6 277.8 1.4Manufacturing - DiscreteUnited States Multiuser Mainframe IBM 88.4
Explain only 15% of total difference as against 90% with compact
Summarizing similar changes
Product Manufacturing - Process
Sum of Revenue YearPlat_Type 1992 1993Other M. 10 473Other S. 0 22Unix M. 7 105Unix S. 1 17Wn16 3 85Wn32 0
PRODUCT GEOGRAPHYPLAT_TYPEPLATFORMYEAR_1992 YEAR_1993 RATIO ERROR(All)- (All)- (All)- (All) 2113.0 2763.5 1.3 200Manufacturing - Process (All) (All) (All) 25.9 702.5 27.1 250Other Vertical Apps (All)- (All)- (All) 20.3 1858.4 91.4 251Other Vertical Apps United StatesUnix S. (All) 8.1 77.5 9.6 0Other Vertical Apps Western EuropeUnix S. (All) 7.3 96.3 13.2 0Manufacturing - Discrete (All) (All) (All) 1135.2 0Health Care (All)- (All)- (All)- 6.9 820.4 118.2 98Banking/Finance United StatesOther M. (All) 341.3 239.3 0.7 60Mechanical CAD United States(All) (All) 327.8 243.4 0.7 34
MDL model for summarization Given N, find the best N rows of answer such that:
if user knows cube-A and answer, number of bits needed to send cube-B is minimized.
Year 1990Products
Geography OS DBMS Prog
Asia 100 80 80
USA 100 200 400
UK 140 100 56Year 1991
ProductsGeography OS DBMS Prog
Asia 90 80 70
USA 89 39 67
UK 140 60 56
N row answer
Cube-A
Cube-B
Integration
Single pass on data --- all indexing/sorting in the DBMS: interactive.
Low memory usage: independent of number of tuples: O(NL)
Easy to package as a stored procedure on the data server side.
When detailed subcube too large: work off aggregated data.
The Relax operator
Example query: generalizing drops
Ratio generalization
The Inform operator
User-cognizant data exploration: overview
Monitor to find regions of data user has visited
Model user’s expectation of unseen values Report most informative unseen values
How to
Model expected values?
Define information content?
Modeling expected values
OS DB Word Prog
Asia 10 10 10 10
Afric 10 10 10 10
USA 10 10 10 10
UK 10 10 10 10
OS DB Word Prog
Asia 5 5 5 5
Afric 20 20 20 20
USA 12 12 12 12
UK 3 3 3 3
OS DB Word Prog
Asia 4 8 7 1
Afric 10 20 30 20
USA 5 9 1 33
UK 1 3 2 6
OS DB Word Prog
AsiaAfricUSAUK
Database hidden from user
Views seen by user
All
All 160
All
Asia 20
Afric 80
USA 48
UK 12
OS DB Word Prog
ALL 20 40 40 60
The Maximum Entropy Principle Choose the most uniform distribution while adhering to
all the constraints E.T.Jaynes..[1990]
it agrees with everything that is known but carefully avoids assuming anything that is not known. It is transcription into mathematics of an ancient principle of wisdom…
Characterizing uniformity:
maximum when all pi-s are equal Solve the constrained optimization problem:
maximize H(p) subject to k constraints
i
ii pppH log)(
Modeling expected values
OS DB Word Prog
Asia 4 8 7 1
Afric 10 20 30 20
USA 5 9 1 33
UK 1 3 2 6
All
All 160
All
Asia 20
Afric 80
USA 48
UK 12
OS DB Word Prog
Asia 5 5 5 5
Afric 20 20 20 20
USA 12 12 12 12
UK 3 3 3 3
OS DB Word Prog
ALL 20 40 40 60
DatabaseVisited views
OS DB Word Prog
Asia 2.5 5 5 7.5
Afric 10 20 20 30
USA 6 12 12 18
UK 1.5 3 3 4.5
Prog
Usa 33
OS DB Word Prog
Asia 3 6 6 4.8
Afric 12 24 24.3 19.2
USA 3 6 6 33
UK 2 4 3.6 3
OS DB Word Prog
Asia 10 10 10 10
Afric 10 10 10 10
USA 10 10 10 10
UK 10 10 10 10
Change in entropy
2
2.1
2.2
2.3
2.4
2.5
2.6
2.7
2.8E
ntr
op
y
View 1 View 2 View 3 View 4 Data
Visited views
Finding expected values Solve the constrained optimization problem:
maximize H(p) subject to k constraints Each constraint is of the form: sum of arbitrary
sets of values Expected values can be expressed as a
product of k coefficients one from each of the k constraints
ki
jIijip
0
)(
Iterative scaling algorithmInitially all p values are the same
While convergence not reached
For each constraint Ci in turn
Scale p values included in Ci by
Converges to optimal solution when all constraints are consistent.
)(
)(~
i
i
Cp
Cp
All
Asia 40
Afric 40
USA 40
UK 40
OS DB Word Prog
ALL 40 40 40 40
Prog
Usa 40
All
Asia 20
Afric 80
USA 48
UK 12
OS DB Word Prog
ALL 20 40 40 60
Prog
Usa 33
OS DB Word Prog
Asia 10 10 10 10
Afric 10 10 10 10
USA 10 10 10 10
UK 10 10 10 10
OS DB Word Prog
ALL 40 40 40 40
Prog
Usa 12
All
Asia 20
Afric 80
USA 48
UK 12
OS DB Word Prog
ALL 20 40 40 60
Prog
Usa 33
OS DB Word Prog
Asia 5 5 5 5
Afric 20 20 20 20
USA 12 12 12 12
UK 3 3 3 3
All
Asia 20
Afric 80
USA 48
UK 12
Prog
Usa 18
All
Asia 20
Afric 80
USA 48
UK 12
OS DB Word Prog
ALL 20 40 40 60
Prog
Usa 33
OS DB Word Prog
Asia 3 5 5 7.5
Afric 10 20 20 30
USA 6 12 12 18
UK 2 3 3 4.5
All
Asia 20
Afric 80
USA 48
UK 12OS DB Word Prog
ALL 20 40 40 60
Prog
Usa 33
All
Asia 20
Afric 80
USA 48
UK 12
OS DB Word Prog
ALL 20 40 40 60
Prog
Usa 33
OS DB Word Prog
Asia 3 5 5 7.5
Afric 10 20 20 30
USA 6 12 12 33
UK 2 3 3 4.5
All
Asia 20
Afric 80
USA 63
UK 12
OS DB Word Prog
ALL 19 37 37 75
Prog
Usa 25
All
Asia 20
Afric 80
USA 48
UK 12
OS DB Word Prog
ALL 20 40 40 60
Prog
Usa 33
OS DB Word Prog
Asia 3 5 5 7.5
Afric 10 20 20 30
USA 5 9 9 25
UK 2 3 3 4.5
All
Asia 20
Afric 80
USA 48
UK 12
OS DB Word Prog
ALL 19 37 37 67
Defined as how much adding it as a constraint will reduce distance between actual and expected values
Distance between actual and expected:
Information content of (k+1)th constraint Ck+1:
Can be approximated as:
Information content of an unvisited cell
i
ki
ii
k
p
ppppD
~log~)~,(
)~,()~,( 1 ppDppD kk
)(
)(~log)(~
1
11
kk
kk Cp
CpCp
Information content of unseen data
0
0.005
0.01
0.015
0.02
0.025
0.03
OS DB Word Prog
AsiaAfric
USAUK
OS DB Word Prog
Asia 4 8 7 1
Afric 10 20 30 20
USA 5 9 1 33
UK 1 3 2 6
OS DB Word Prog
Asia 3 6 6 4.8
Afric 12 24 24.3 19.2
USA 3 6 6 33
UK 2 4 3.6 3
Finding N most informative cells
In general, most informative cells can be any of value from any level of aggregation.
Single-pass algorithm that finds the best difference between actual and expected values [Diff algorithm]
Information gain with focussed exploration
0
0.2
0.4
0.6
0.8
1
0 10 20 30 40 50
Constraint number
Rela
tive s
quare
err
or Random MaxEntropy
0
0.2
0.4
0.6
0.8
1
0 10 20 30 40 50
Constraint number
Rela
tive s
quare
err
or
Random MaxEntropy
0
0.1
0.2
0.3
0.4
0 5 10 15
Constraint number
Rel
ativ
e sq
uare
err
or
Illustration from Student enrollment data
Student Sex Program Department YearCategory (9) Sex (2) Name (10) Name (28) Year (10) Category (3)
Sum 8206
PROGRAM Total2 Yr M.Sc 9.21%B.Tech 33.87%M.Tech 37.31%Ph.D 11.60%Others 1.60%
SEX TotalF 10.25%M 89.75%
CATEGORY TotalFull time Sponsored 5.28%Indian 81.55%Others 1.90%
Year Total1989 19.00%
Others 9.00%
35% of information in data captured in 12 out of 4560 cells: 0.25% of data
Top few suprising values
Category Sex Program DeptIndian M Computer Science & EngineeringIndian M Metallurgical Engineering & Mat.Sc.
M.Mgnt. School of ManagementIndian M M.Tech Civil EngineeringIndian M M.Tech Chemical Engineering
M Bio-Technology
80% of information in data captured in 50 out of 4560 cells: 1% of data
Summary
Our goal: enhance OLAP with a suite of operations that are richer than simple OLAP and SQL queries more interactive than conventional mining
...and thus reduce the need for manual analysis
Proposed three new operators: Diff, Relax,Inform Formulations with theoretical basis Efficient algorithms for online answering Integrates smoothly with existing systems.