cse 8392 spring 1999 data mining: part i professor margaret h. dunham department of computer science...
Post on 12-Jan-2016
221 Views
Preview:
TRANSCRIPT
CSE 8392 SPRING 1999DATA MINING:
PART I
Professor Margaret H. Dunham
Department of Computer Science and Engineering
Southern Methodist University
Dallas, Texas 75275
(214) 768-3087
fax: (214) 768-3085
email: mhd@seas.smu.edu
www: http://www.seas.smu.edu/~mhd
January 1999
CSE 8392 Spring 1999 2
CSE8392 SPRING 1999 OUTLINE
• Course Objective: To examine Data Mining concepts. A database perspective (rather than AI or statistics) is taken.
• I. Introduction and Related Topics
• II. Core Topics
• III. Advanced Topics
• IV. Case Studies
• V. Student Presentations
• VI. Summary and Future Trends
CSE 8392 Spring 1999 3
INTRODUCTION AND RELATED TOPICS
• Section Objective: Provide an introduction of data mining concepts. Briefly examine related concepts and background topics.
• Historical Perspective
– Gleaning Knowledge from the Data
– User Expectations increase as amount/sophistication of collected data increases.
– Reality vs Extracted Data
Reality
QueryInformation
Need
Data
Physical View Database View
CSE 8392 Spring 1999 4
Related Topics (to be covered)
– Knowledge Discovery
– Information Retrieval
– Fuzzy Sets
– Data Warehousing and OLAP
– Dimensional Modeling
CSE 8392 Spring 1999 5
Data Mining Overview
• What is Data Mining?– Definition: Fayyad, p. 9 – A.k.a.
• Exploratory data analysis• Unsupervised pattern recognition• Data driven discovery• Deductive learning
• Data Mining determines patterns in the data– Non-trivial– Valid– Novel– Potentially useful– Interesting– General and simple– Understandable
CSE 8392 Spring 1999 6
DM Techniques (R[1])
• DM involves many different algorithms to accomplish different things. All have the following techniques in common.
– Model(Must fit a model to the data.)
• Function/Purpose
• Representation
– Preference Criteria (How to choose one model over another?)
– Search Algorithm (How to search the data)
• Example (Loan Data, fig 1.1 p6 in Fayyad):
– Model: Classification, Linear Function
– Preference: What best fits data? (Fig 1.2 or 1.4)
– Search Algorithm: Linear search of database
CSE 8392 Spring 1999 7
DM Model Functions (R[1])
• Classification - Map data into predefined groups
• Regression - Map data to real valued predicate variable
• Clustering - Map data into groups defined by data itself
• Summarization - Map subsets of data into simple description
• Dependency Modeling - Identify dependencies among data items
• Link Analysis - Identify other relationships among data (association rules)
• Sequence Analysis - Identify sequential patterns in data
CSE 8392 Spring 1999 8
DM Historical Perspective
• Late 70’s: Spreadsheet analysis
• 80’s: Transactional databases support data storage and retrieval
• Early 90’s: Growing interest in end user support (a.k.a. decision support)
– Issue: transactional databases are not designed for decision support
• Mid 90’s: Dedicated data warehouses for decision support and multidimensional analysis
• Late 90’s: Proliferation; new concepts (data marts)
• DM Tools: Neovista, Red Brick
CSE 8392 Spring 1999 9
Data Mining Metrics
• Berson, Tables 17-1,17-2,17-3, p 347
• Accuracy
• Clarity
• Dirty Data
• Dimensionality
• Raw Data (Preprocessing)
• RDBMS embedding
• Scalability
• Speed
• Validation
CSE 8392 Spring 1999 10
DM Issues
• Overfitting
• Outliers
• Closed World Assumption
• Database schemas and database models
• Algorithms for data mining
• Interpretation and visualization of results
• Size of databases
• Multimedia data, Spatio-Temporal Data
• Changing data
• Integration
• DM Applications
– Basket market analysis Stock analysis and selection
– Fraud detection and prevention
– Crisis prediction and prevention
CSE 8392 Spring 1999 11
KNOWLEDGE DISCOVERY IN DATABASES (KDD)
• “Overall process of discovering useful knowledge from data.” (p28 in R[1])
• Defn: R[1] p 30
• Steps Fig 1, p29 R[1] (Fig 1.3 in Fayyad)
• Data Mining is one step in KDD process
• KDD objective not usually clear or exact. May require time with customer understanding needs.
• Data usually has problems - needs cleaning
– Incorrect/missing data
– Extract from multiple sources and compare
– Delete anomalous data and sources
– Different data types/metrics
CSE 8392 Spring 1999 12
FUZZY SETS and LOGIC
• Set membership described by a real valued (0,1) membership function
• Ex: Set of all tall people
• Set membership function: f(x)=x is tall iff height(x)>6 ft.
• Note that this is a simple classification problem. Just as the Loan example, the results are not exact.
• Basis of many classification and clustering approaches
• In a conventional DB how do you retrieve all tall people?
– Three valued logic: True, False, Maybe
– Multi-valued logic: More than 2 values
CSE 8392 Spring 1999 13
Fuzzy Logic
• Reasoning with uncertainty
• Extends multivalued logic; allows user to communicate using imprecise concepts, i.e.
– “good” and “bad”
– “close to” and “far away”
• Avoids brittleness of rule based reasoning by introducing probability of set membership
– Allows for smoother transition between classification sets in the domain
– Example
• Berson figure 16.2, page 325
CSE 8392 Spring 1999 14
INFORMATION RETRIEVAL
• Store and retrieve documents based on fuzzy queries
• Predecessor of web based access
• Ex: Store information about all articles in all IEEE Transactions journals and Retrieve all documents dealing with heaps.
• Overview
– Conventional IR Systems
– Query Structures(Keywords)
– Matching(Multivalued logic)
– Measures
– Text Analysis Techniques
– IR Related Topics
CSE 8392 Spring 1999 15
Conventional IR Systems
• Library card catalogs
• Documents (Library Science)
– Formatted
– Unformatted (Text)
– Mixed
• Document Surrogates
– Identifiers
– Titles, names, and dates
– Abstracts, extracts, reviews
– Summaries of Numerical Data
– Image Descriptions
CSE 8392 Spring 1999 16
IR Queries
• Query Structures
– Matching Criteria
– Boolean Queries
– Vector
– Fuzzy
– Natural Language
• Logical combination of keywords
• Weight associated with keywords
• Similarity measures
CSE 8392 Spring 1999 17
Similarity Measures
– Document Vector:
– Different Measures:
– Salton and McGill, Introduction to Modern Information Retrieval, 1984, McGraw-Hill, pp201-204.
– Similarity uses:
• Document-Document
• Query-Query
• Document-Query
iniii dddD ,...,, 21
n
kjkikji ddDDSim
1
),(
CSE 8392 Spring 1999 18
IR Document/Query Matching
• Matching Process
– Relevance and Similarity Measures
– Boolean based matching
• Logical match
– Vector based matching
• Threshold match
– Probabilistic Match
n documents relevant
• P(relevant) =
N total documents
– Fuzzy Matching
– Proximity Matching
– Weighting
– Relative Importance of Items
CSE 8392 Spring 1999 19
IR Matching
• Scaling
– Impact of Sample Size
– Clustering
– Centroids
• Measures
– Precision
– Recall
CSE 8392 Spring 1999 20
IR Indexing
• Text Analysis
– Indexing is the assignment of keywords or terms that represent document content
• Originally a library science problem that has grown with the advent of web based searches
– Indexing types
• Automated vs. manual
• Controlled vs. uncontrolled
• Single term vs. terms in context
• Deep vs. shallow
CSE 8392 Spring 1999 21
IR Indexing
• General Steps
– 1. Assignment of terms or concepts capable of representing content
– 2. Assignment to each term a weight or value
• Indexing
– Vector based
• Start with excerpts, remove high frequency words
– Stop list
– Thesaurus
• Compute discrimination values of terms
CSE 8392 Spring 1999 22
IR Retrieval
• Retrieval or Classification
– Vector based
• Same starting point as with indexing
• Compute weighting factors
• Assign to each document a weighted term vector
– Similarity Measures
• Measure similarity between document/query
• Results normalized to range between 0 - 1
CSE 8392 Spring 1999 23
IR Retrieval
– Inverse Document Frequency
• Assumes importance is proportional to standard occurrence frequency, and inversely proportional to the total number of documents.
• Also used for similarity measurement
– Inverted Indexing of Document
– Concept Hierarchy
• DAG of concepts
• Follow nodes from general to more specific
• Tag articles with low level concepts so that each may be distinguished from ancestors
CSE 8392 Spring 1999 24
IR Related Topics
• Information Retrieval Related Topics
– Text Analysis
– Fuzzy Sets
– Extending Databases
– Hypertext
– Digital Libraries
– Data Mining
• Web based browsers
CSE 8392 Spring 1999 25
DATA WAREHOUSING AND OLAP
– Preparations for Mining: Data Warehousing
• Extracting the data (from RDBMS)
• Storing the data
– Data warehouse or data mart
• Cleansing the data
• Mining the data
– Often with multidimensional queries
• Definition
– Blend of technologies
– Integration
– Enables Strategic Use of Data
• Architecture
– Figure 6.1, page 116
CSE 8392 Spring 1999 26
DW Migration
• Migration from Relational Database to Data Warehouse
– Differences (Relational vs. Data Warehouse)
– Procedure for Migration
• Extraction
• Cleanup
• Transformation
• Migration
• Issues
– Multiple sources
– Database Heterogeneity
– Data Heterogeneity
CSE 8392 Spring 1999 27
DW Design
• Data Warehouse Design Considerations - Nine Step Method:
– Subject Matter
– Fact Table contents
– Dimensioning
– Fact Selection
– Precalculations
– Rounding out dimension table
– Duration selection
– What about change?
– Query priorities
• Technical Considerations
– Hardware
– Communications Infrastructure
– Data Structures
CSE 8392 Spring 1999 28
More on DW• Benefits
– Development of strategic information and resources
– Hypothesis testing
– Knowledge discovery
• Data Marts
– Definition: a mini data warehouse for data mining
– Directed at a partition of data
– Dedicated user group
– May be physically separate
– Drivers
• Urgent user requirements
• Small budget
• Absence of sponsor
• Decentralization
• Smaller project size
CSE 8392 Spring 1999 29
DIMENSIONAL MODELING
• Dimensional Modeling– Describes relationships in the data that
will be mined– Relatively new concept, still developing– A technique for visualizing data models– Schema (Star and Snowflake)– Facts - A collection of related data items,
consisting of measures and context data– Dimensions - A collection of members or
units of the same type of view. Axis for modeling. Sets the context for the facts.
– Measures - Numeric attribute of fact (What is stored about sales data)
• Focus - Tends to be on numeric data• MD Analysis vs. DM - Figure 4, R[3]
CSE 8392 Spring 1999 30
Data Cube
• Way to visualize facts and dimensions
• Hypercube (more than 3 dimensions)
• May be nested
• Figure 13.1, p249, Berson
• Figure 15,R[3]
CSE 8392 Spring 1999 31
Part No.
Dimension
Customer
Dimension
Time
Dimension
Salesperson
Dims
Product
Dimension
Sales
Facts
Star Schema
– Contains large fact table and a surrounding set of dimension tables
– A.k.a. constellation or multistar model
– Figure 9.1, p171,Berson
– Following from Figure 18, R[3]
CSE 8392 Spring 1999 32
Part No.
Dimension
Customer
Dimension
Time
Dimension
Salesperson
Dimension
Product
Dimension
Sales
Facts
Location
Dimension
Manager
Dimension
Month
Dimension
Week
Dimension
Snowflake Schema
• Sometimes dimensions have hierarchies among themselves
• N:1 relationships among members of a dimension may be subdivided
• Decomposition yields a snowflake like schema
CSE 8392 Spring 1999 33
OLAP (On Line Analytic Processing)• Multidimensional database• Allows user to analyze data using elaborate,
multidimensional, complex views• MOLAP - Multidimensional OLAP.
Supported by specialized DBMS/software systems. (Data structures, temporal)– May not be general enough for other uses– Access limited and optimized for OLAP
processing– Fig 13.3 p 253, Berson
• ROLAP - Underlying data stored in traditional (relational) DBMS and accessed by traditional query language (SQL).– Layer on top of DBMS. Middleware.– May have poor performance for OLAP
applications– Fig 13.4 p 254, Berson
CSE 8392 Spring 1999 34
OLAP Operations
• Move view of facts down/up dimensions
– Drill Down
– Roll Up
– Figure 3, R[3]
– Figure 16,R[3]
• Look at data by partitioning the cube
– Slice - Look at subcube to get more specific data
– Dice - Rotate cube to look at another dimension
– Figure 17,R[3]
top related