When to use Data Mining

Introduction

• An important question that should be answered before you commence any data mining project is whether data mining techniques are, in fact necessary.

• In determining this it is important to understand what level of sophistication of data mining is required. For instance, do you just need a few standardized printed reports or do you need interactive ROI analysis or OLAP analysis to see what your data looks like?

• Do you need or true data mining techniques that build predictive models to search through your database for useful patterns?

The Data Mining Process

What all Data Mining techniques have in common

• Each Data Mining algorithm has the following in common:– Model Structure. The structure that defines the model

(Is it a tree, a neural network, or a neighbor?)

– Search. How does the algorithm amend and modify the model over time as more data is made available

– Validation. When does the algorithm terminate because it has created a valid model?

What all Data Mining techniques have in common (cont’d)

Data Mining in the Business Process

• When Data Mining is used for non-exploratory reasons or whenever supervised learning techniques are used, this customer reaction provide a fairly well-defined target column within the database, which relates to the business process. The target must have the following attributes in order to be successful with data mining:– The target has value– The target is actionable– The effect of action can be captured

Data Mining in the Business Process (cont’d)

Avoiding some big mistakes in Data Mining

• The technology-centered view of the data mining process emphasizes getting the model right, with the assumption that the predictive product has been well-defined and that the data that has been captured to date is well understood.

• This is not always the case.

Three measures for Data Mining Tools

• Accuracy. The data mining tool must produce a model that is as accurate as possible.

• Explanation. The data mining tool needs to be able to ‘explain’ how the model works to the end user in a clear way

• Integration. The data mining tool must integrate with the current business process, and data and information flow in the company.

• When these three requirements are well met, the data mining tools will produce highly profitable models that are likely to remain stable over long periods of time.

Embedded Data Mining for business

How to measure Accuracy, Explanation, and Integration

• Measuring Accuracy:– Accuracy– Error rate– Error rate at rejection– Mean squared error– Lift– Profit/ROI

How to measure Accuracy, Explanation, and Integration

• Measuring Explanation:– Automated rule generation– OLAP integration– Model validation

• Measuring Integrity– Proprietary data extracts– Metadata– Predictor preprocessing– Predictor/prediction types– Dirty data– Missing values– Scalability

What the Future holds for Embedded Data Mining

• Once the data mining process becomes easy enough to use and is seamlessly integrated into business process and the general data and information flow around the enterprise, there will be new applications and synergies that will make data mining an even more critical requirement for any fully functioning data warehouse– Use data mining to improve the multidimensional database

– Use data mining to improve the data warehouse structure

– Multidimensional databases and summary data will enhance data mining performance. The more data, the better any data mining technique is