data warehousing, data mining and web warehouses
DESCRIPTION
TRANSCRIPT
![Page 1: Data Warehousing, Data Mining and Web Warehouses](https://reader033.vdocuments.net/reader033/viewer/2022061217/54b418804a7959645b8b45b8/html5/thumbnails/1.jpg)
Slide 1
COMM1E Lecture Eleven
Data Warehousing, Mining and Web Data Warehousing, Mining and Web ToolsTools
![Page 2: Data Warehousing, Data Mining and Web Warehouses](https://reader033.vdocuments.net/reader033/viewer/2022061217/54b418804a7959645b8b45b8/html5/thumbnails/2.jpg)
Slide 2
COMM1E Lecture Eleven
ContentsContents
• Data Warehousing• Data Mining• Web Warehouses• Further Reading
![Page 3: Data Warehousing, Data Mining and Web Warehouses](https://reader033.vdocuments.net/reader033/viewer/2022061217/54b418804a7959645b8b45b8/html5/thumbnails/3.jpg)
Slide 3
COMM1E Lecture Eleven
OLTP SystemsOLTP Systems
• So far we have concentrated on OLTP (online transaction processing) systems – range in size from megabytes to terabytes
– high transaction throughput
• Decision makers require access to all data wherever it is located– current data
– historical data
![Page 4: Data Warehousing, Data Mining and Web Warehouses](https://reader033.vdocuments.net/reader033/viewer/2022061217/54b418804a7959645b8b45b8/html5/thumbnails/4.jpg)
Slide 4
COMM1E Lecture Eleven
OLTP SystemsOLTP Systems
• Holds current data• Stores detailed data• Data is dynamic• Repetitive processing• High level of transaction throughput• Predictable pattern of usage• Transaction driven• Application-oriented• Supports day-to-day decisions• Serves large number of clerical/operational users
![Page 5: Data Warehousing, Data Mining and Web Warehouses](https://reader033.vdocuments.net/reader033/viewer/2022061217/54b418804a7959645b8b45b8/html5/thumbnails/5.jpg)
Slide 5
COMM1E Lecture Eleven
Data Warehouse DefinitionData Warehouse Definition
• ‘A data warehouse is a – subject-oriented,
– integrated,
– time-variant and
– non-volatile
• collection of data in support of management’s decision-making process’ (Inmon 1993)
![Page 6: Data Warehousing, Data Mining and Web Warehouses](https://reader033.vdocuments.net/reader033/viewer/2022061217/54b418804a7959645b8b45b8/html5/thumbnails/6.jpg)
Slide 6
COMM1E Lecture Eleven
Data Warehousing SystemsData Warehousing Systems
• Holds historical data• Stores detailed, lightly and highly summarised
data• Data is largely static• Ad-hoc, unstructured and heuristic processing• Medium/low level of transaction throughput• Unpredictable pattern of usage• Analysis driven• Subject-oriented• Supports strategic decisions• Serves relatively low no. of managerial users
![Page 7: Data Warehousing, Data Mining and Web Warehouses](https://reader033.vdocuments.net/reader033/viewer/2022061217/54b418804a7959645b8b45b8/html5/thumbnails/7.jpg)
Slide 7
COMM1E Lecture Eleven
BenefitsBenefits
• Potential high returns on investment– 401% return of investment (over three years) for 90% of
companies in 1996
• Competitive advantage– data can reveal previously unknown, unavailable and
untapped information
• Increased productivity of corporate decision-makers– integration allows more substantive, accurate and
consistent analysis
![Page 8: Data Warehousing, Data Mining and Web Warehouses](https://reader033.vdocuments.net/reader033/viewer/2022061217/54b418804a7959645b8b45b8/html5/thumbnails/8.jpg)
Slide 8
COMM1E Lecture Eleven
ArchitectureArchitecture
Warehouse mgr
Loadmgr
Warehouse mgr
Querymanager
DBMS
Meta-data Highlysummarizeddata
Lightly summarizeddata
Detailed data
Mainframe operationaln/w,h/w data
DepartmentalRDBMS data
Private data
External dataArchive/backup
Reporting, query,application development,EIS tools
OLAP tools
Data-mining tools
![Page 9: Data Warehousing, Data Mining and Web Warehouses](https://reader033.vdocuments.net/reader033/viewer/2022061217/54b418804a7959645b8b45b8/html5/thumbnails/9.jpg)
Slide 9
COMM1E Lecture Eleven
Information FlowsInformation Flows
Warehouse Mgr
Loadmgr
Warehouse mgr
Querymanager
DBMS
Meta-data
Highlysumm.data
Lightlysumm.
Detailed data
Operational datasource 1
Operational datasource n Archive/backup
Reporting query, appdevelopment,EIS tools
OLAP tools
Data-mining tools
Meta-flow
Inflow
Downflow
Upflow
Outflow
![Page 10: Data Warehousing, Data Mining and Web Warehouses](https://reader033.vdocuments.net/reader033/viewer/2022061217/54b418804a7959645b8b45b8/html5/thumbnails/10.jpg)
Slide 10
COMM1E Lecture Eleven
Information Flow ProcessesInformation Flow Processes
• Five primary information flows– Inflow - extraction, cleansing and loading of data from
source systems into warehouse
– Upflow - adding value to data in warehouse through summarizing, packaging and distributing data
– Downflow - archiving and backing up data in warehouse
– Outflow - making data available to end users
– Metaflow - managing the metadata
![Page 11: Data Warehousing, Data Mining and Web Warehouses](https://reader033.vdocuments.net/reader033/viewer/2022061217/54b418804a7959645b8b45b8/html5/thumbnails/11.jpg)
Slide 11
COMM1E Lecture Eleven
Data Warehouse DesignData Warehouse Design
• Data must be designed to allow ad-hoc queries to be answered with acceptable performance constraints
• Queries usually require access to factual data generated by business transactions– e.g. find the average number of properties rented out with
a monthly rent greater than £700 at each branch office over the last six months
• Uses Dimensionality Modelling
![Page 12: Data Warehousing, Data Mining and Web Warehouses](https://reader033.vdocuments.net/reader033/viewer/2022061217/54b418804a7959645b8b45b8/html5/thumbnails/12.jpg)
Slide 12
COMM1E Lecture Eleven
Dimensionality ModellingDimensionality Modelling
• Similar to E-R modelling but with constraints– composed of one fact table with a composite primary
key
– dimension tables have a simple primary key which corresponds exactly to one foreign key in the fact table
– uses surrogate keys based on integer values
– Can efficiently and easily support ad-hoc end-user queries
![Page 13: Data Warehousing, Data Mining and Web Warehouses](https://reader033.vdocuments.net/reader033/viewer/2022061217/54b418804a7959645b8b45b8/html5/thumbnails/13.jpg)
Slide 13
COMM1E Lecture Eleven
Star SchemasStar Schemas
• The most common dimensional model• A fact table surrounded by dimension tables• Fact tables
– contains FK for each dimension table– large relative to dimension tables– read-only
• Dimension tables– reference data– query performance can be speeded up by denormalising
into a single dimension table
![Page 14: Data Warehousing, Data Mining and Web Warehouses](https://reader033.vdocuments.net/reader033/viewer/2022061217/54b418804a7959645b8b45b8/html5/thumbnails/14.jpg)
Slide 14
COMM1E Lecture Eleven
E-R Model ExampleE-R Model Example
![Page 15: Data Warehousing, Data Mining and Web Warehouses](https://reader033.vdocuments.net/reader033/viewer/2022061217/54b418804a7959645b8b45b8/html5/thumbnails/15.jpg)
Slide 15
COMM1E Lecture Eleven
Star Schema ExampleStar Schema Example
![Page 16: Data Warehousing, Data Mining and Web Warehouses](https://reader033.vdocuments.net/reader033/viewer/2022061217/54b418804a7959645b8b45b8/html5/thumbnails/16.jpg)
Slide 16
COMM1E Lecture Eleven
Data MiningData Mining
• ‘The process of extracting valid, previously unknown, comprehensible and actionable information from large databases and using it to make crucial business decisions’– focus is to reveal information which is hidden or
unexpected– patterns and relationships are identified by examining
the underlying rules and features of the data– work from data up– require large volumes of data
![Page 17: Data Warehousing, Data Mining and Web Warehouses](https://reader033.vdocuments.net/reader033/viewer/2022061217/54b418804a7959645b8b45b8/html5/thumbnails/17.jpg)
Slide 17
COMM1E Lecture Eleven
Example Data Mining ApplicationsExample Data Mining Applications
• Retail/Marketing– Identifying buying patterns of customers
– Finding associations among customer demographic characteristics
– Predicting response to mailing campaigns
– Market basket analysis
![Page 18: Data Warehousing, Data Mining and Web Warehouses](https://reader033.vdocuments.net/reader033/viewer/2022061217/54b418804a7959645b8b45b8/html5/thumbnails/18.jpg)
Slide 18
COMM1E Lecture Eleven
Example Data Mining ApplicationsExample Data Mining Applications
• Banking– Detecting patterns of fraudulent credit card use
– Identifying loyal customers
– Predicting customers likely to change their credit card affiliation
– Determining credit card spending by customer groups
![Page 19: Data Warehousing, Data Mining and Web Warehouses](https://reader033.vdocuments.net/reader033/viewer/2022061217/54b418804a7959645b8b45b8/html5/thumbnails/19.jpg)
Slide 19
COMM1E Lecture Eleven
Data Mining TechniquesData Mining Techniques
• Predictive Modelling– using observations to form a model of the important
characteristics of some phenomenon
• Techniques:– Classification
– Value Prediction
![Page 20: Data Warehousing, Data Mining and Web Warehouses](https://reader033.vdocuments.net/reader033/viewer/2022061217/54b418804a7959645b8b45b8/html5/thumbnails/20.jpg)
Slide 20
COMM1E Lecture Eleven
Classification Example: Tree Classification Example: Tree InductionInduction
Customer renting property> 2 years
Rent property
Rent property Buy property
Customer age> 25 years?
No Yes
No Yes
![Page 21: Data Warehousing, Data Mining and Web Warehouses](https://reader033.vdocuments.net/reader033/viewer/2022061217/54b418804a7959645b8b45b8/html5/thumbnails/21.jpg)
Slide 21
COMM1E Lecture Eleven
Data Mining TechniquesData Mining Techniques
• Database Segmentation:– to partition a database into an unknown number of
segments (or clusters) of records which share a number of properties
• Techniques:– Demographic clustering
– Neural clustering
![Page 22: Data Warehousing, Data Mining and Web Warehouses](https://reader033.vdocuments.net/reader033/viewer/2022061217/54b418804a7959645b8b45b8/html5/thumbnails/22.jpg)
Slide 22
COMM1E Lecture Eleven
Database Segmentation: Database Segmentation: Scatterplot ExampleScatterplot Example
![Page 23: Data Warehousing, Data Mining and Web Warehouses](https://reader033.vdocuments.net/reader033/viewer/2022061217/54b418804a7959645b8b45b8/html5/thumbnails/23.jpg)
Slide 23
COMM1E Lecture Eleven
Data Mining TechniquesData Mining Techniques
• Link Analysis– establish associations between individual records (or
sets of records) in a database• e.g. ‘when a customer rents property for more than two years
and is more than 25 year olds, then in 40% of cases, the customer will buy the property’
– Techniques
– Association discovery
– Sequential pattern discovery
– Similar time sequence discovery
![Page 24: Data Warehousing, Data Mining and Web Warehouses](https://reader033.vdocuments.net/reader033/viewer/2022061217/54b418804a7959645b8b45b8/html5/thumbnails/24.jpg)
Slide 24
COMM1E Lecture Eleven
Data Mining TechniquesData Mining Techniques
• Deviation Detection– identify ‘outliers’, something which deviates from
some known expectation or norm
– Statistics
– Visualisation
![Page 25: Data Warehousing, Data Mining and Web Warehouses](https://reader033.vdocuments.net/reader033/viewer/2022061217/54b418804a7959645b8b45b8/html5/thumbnails/25.jpg)
Slide 25
COMM1E Lecture Eleven
Deviation Detection: Visualisation Deviation Detection: Visualisation ExampleExample
![Page 26: Data Warehousing, Data Mining and Web Warehouses](https://reader033.vdocuments.net/reader033/viewer/2022061217/54b418804a7959645b8b45b8/html5/thumbnails/26.jpg)
Slide 26
COMM1E Lecture Eleven
Mining and WarehousingMining and Warehousing
• Data mining needs single, separate, clean, integrated, self-consistent data source
• Data warehouse well equipped:– populated with clean, consistent data
– contains multiple sources
– utilizes query capabilities
– capability to go back to data source
![Page 27: Data Warehousing, Data Mining and Web Warehouses](https://reader033.vdocuments.net/reader033/viewer/2022061217/54b418804a7959645b8b45b8/html5/thumbnails/27.jpg)
Slide 27
COMM1E Lecture Eleven
Web WarehousesWeb Warehouses
• The ultimate data warehouse is the Internet– contains data in numerous formats
• relational
• object-oriented
• semi-structured
• unstructured ...
• It is impossible to store all this data in a warehouse– imagine the storage required!
– See Internet Joke – http://www.w3schools.com
• So need an intermediary
![Page 28: Data Warehousing, Data Mining and Web Warehouses](https://reader033.vdocuments.net/reader033/viewer/2022061217/54b418804a7959645b8b45b8/html5/thumbnails/28.jpg)
Slide 28
COMM1E Lecture Eleven
XMLXML
• A meta-language that enables designers to create their own customised tags to provide functionality not available within HTML
• e.g.<STAFF>
<NAME>
<FNAME>John</FNAME><LNAME>White</LNAME>
</NAME>
<SEX gender=‘M’/>
</STAFF>
![Page 29: Data Warehousing, Data Mining and Web Warehouses](https://reader033.vdocuments.net/reader033/viewer/2022061217/54b418804a7959645b8b45b8/html5/thumbnails/29.jpg)
Slide 29
COMM1E Lecture Eleven
XML ToolsXML Tools
• Can define stylesheets to display XML database in web pages
• Can write queries:WHERE <STAFF><GENDER>$$</GENDER><NAME><FNAME>$F</FNAME><LNAME>$L</LNAME></NAME>$$ = ‘M’CONSTRUCT <LNAME>$L</LNAME>
• To build a warehouse can develop a representation of data models in XML
• Good as a common format for EDI
![Page 30: Data Warehousing, Data Mining and Web Warehouses](https://reader033.vdocuments.net/reader033/viewer/2022061217/54b418804a7959645b8b45b8/html5/thumbnails/30.jpg)
Slide 30
COMM1E Lecture Eleven
Further ReadingFurther Reading
• Connolly and Begg, chapters 30, 31 and 32.• W H Inmon, Building the Data Warehouse, New
York, Wiley and Sons, 1993.• Benyon-Davies P, Database Systems (2nd ed.),• York, Wiley and Sons, 1993.• White Paper on Global, XML Repositories for
XML/EDI. – http://ww.xmledi.com/repository/xml-repWP.htm