■ Data mining and knowledge discovery indatabases have been attracting a significantamount of research, industry, and media atten-tion of late. What is all the excitement about?This article provides an overview of this emergingfield, clarifying how data mining and knowledgediscovery in databases are related both to eachother and to related fields, such as machinelearning, statistics, and databases. The articlementions particular real-world applications,specific data-mining techniques, challenges in-volved in real-world applications of knowledgediscovery, and current and future research direc-tions in the field.

Across a wide variety of fields, data arebeing collected and accumulated at adramatic pace. There is an urgent need

for a new generation of computational theo-ries and tools to assist humans in extractinguseful information (knowledge) from therapidly growing volumes of digital data.These theories and tools are the subject of theemerging field of knowledge discovery indatabases (KDD).

At an abstract level, the KDD field is con-cerned with the development of methods andtechniques for making sense of data. The basicproblem addressed by the KDD process is oneof mapping low-level data (which are typicallytoo voluminous to understand and digest easi-ly) into other forms that might be more com-pact (for example, a short report), more ab-stract (for example, a descriptiveapproximation or model of the process thatgenerated the data), or more useful (for exam-ple, a predictive model for estimating the val-ue of future cases). At the core of the process isthe application of specific data-mining meth-ods for pattern discovery and extraction.1

This article begins by discussing the histori-cal context of KDD and data mining and theirintersection with other related fields. A briefsummary of recent KDD real-world applica-tions is provided. Definitions of KDD and da-ta mining are provided, and the general mul-tistep KDD process is outlined. This multistepprocess has the application of data-mining al-gorithms as one particular step in the process.The data-mining step is discussed in more de-tail in the context of specific data-mining al-gorithms and their application. Real-worldpractical application issues are also outlined.Finally, the article enumerates challenges forfuture research and development and in par-ticular discusses potential opportunities for AItechnology in KDD systems.

Why Do We Need KDD?The traditional method of turning data intoknowledge relies on manual analysis and in-terpretation. For example, in the health-careindustry, it is common for specialists to peri-odically analyze current trends and changesin health-care data, say, on a quarterly basis.The specialists then provide a report detailingthe analysis to the sponsoring health-care or-ganization; this report becomes the basis forfuture decision making and planning forhealth-care management. In a totally differ-ent type of application, planetary geologistssift through remotely sensed images of plan-ets and asteroids, carefully locating and cata-loging such geologic objects of interest as im-pact craters. Be it science, marketing, finance,health care, retail, or any other field, the clas-sical approach to data analysis relies funda-mentally on one or more analysts becoming

Articles

FALL 1996 37

From Data Mining toKnowledge Discovery in

DatabasesUsama Fayyad, Gregory Piatetsky-Shapiro, and Padhraic Smyth

Copyright © 1996, American Association for Artificial Intelligence. All rights reserved. 0738-4602-1996 / $2.00

areas is astronomy. Here, a notable successwas achieved by SKICAT, a system used by as-tronomers to perform image analysis,classification, and cataloging of sky objectsfrom sky-survey images (Fayyad, Djorgovski,and Weir 1996). In its first application, thesystem was used to process the 3 terabytes(1012 bytes) of image data resulting from theSecond Palomar Observatory Sky Survey,where it is estimated that on the order of 109

sky objects are detectable. SKICAT can outper-form humans and traditional computationaltechniques in classifying faint sky objects. SeeFayyad, Haussler, and Stolorz (1996) for a sur-vey of scientific applications.

In business, main KDD application areasincludes marketing, finance (especially in-vestment), fraud detection, manufacturing,telecommunications, and Internet agents.

Marketing: In marketing, the primary ap-plication is database marketing systems,which analyze customer databases to identifydifferent customer groups and forecast theirbehavior. Business Week (Berry 1994) estimat-ed that over half of all retailers are using orplanning to use database marketing, andthose who do use it have good results; for ex-ample, American Express reports a 10- to 15-percent increase in credit-card use. Anothernotable marketing application is market-bas-ket analysis (Agrawal et al. 1996) systems,which find patterns such as, “If customerbought X, he/she is also likely to buy Y andZ.” Such patterns are valuable to retailers.

Investment: Numerous companies use da-ta mining for investment, but most do notdescribe their systems. One exception is LBSCapital Management. Its system uses expertsystems, neural nets, and genetic algorithmsto manage portfolios totaling $600 million;since its start in 1993, the system has outper-formed the broad stock market (Hall, Mani,and Barr 1996).

Fraud detection: HNC Falcon and NestorPRISM systems are used for monitoring credit-card fraud, watching over millions of ac-counts. The FAIS system (Senator et al. 1995),from the U.S. Treasury Financial Crimes En-forcement Network, is used to identify finan-cial transactions that might indicate money-laundering activity.

Manufacturing: The CASSIOPEE trou-bleshooting system, developed as part of ajoint venture between General Electric andSNECMA, was applied by three major Euro-pean airlines to diagnose and predict prob-lems for the Boeing 737. To derive families offaults, clustering methods are used. CASSIOPEE

received the European first prize for innova-

intimately familiar with the data and servingas an interface between the data and the usersand products.

For these (and many other) applications,this form of manual probing of a data set isslow, expensive, and highly subjective. Infact, as data volumes grow dramatically, thistype of manual data analysis is becomingcompletely impractical in many domains.Databases are increasing in size in two ways:(1) the number N of records or objects in thedatabase and (2) the number d of fields or at-tributes to an object. Databases containing onthe order of N = 109 objects are becoming in-creasingly common, for example, in the as-tronomical sciences. Similarly, the number offields d can easily be on the order of 102 oreven 103, for example, in medical diagnosticapplications. Who could be expected to di-gest millions of records, each having tens orhundreds of fields? We believe that this job iscertainly not one for humans; hence, analysiswork needs to be automated, at least partially.

The need to scale up human analysis capa-bilities to handling the large number of bytesthat we can collect is both economic and sci-entific. Businesses use data to gain competi-tive advantage, increase efficiency, and pro-vide more valuable services to customers.Data we capture about our environment arethe basic evidence we use to build theoriesand models of the universe we live in. Be-cause computers have enabled humans togather more data than we can digest, it is on-ly natural to turn to computational tech-niques to help us unearth meaningful pat-terns and structures from the massivevolumes of data. Hence, KDD is an attempt toaddress a problem that the digital informa-tion era made a fact of life for all of us: dataoverload.

Data Mining and KnowledgeDiscovery in the Real World

A large degree of the current interest in KDDis the result of the media interest surroundingsuccessful KDD applications, for example, thefocus articles within the last two years inBusiness Week, Newsweek, Byte, PC Week, andother large-circulation periodicals. Unfortu-nately, it is not always easy to separate factfrom media hype. Nonetheless, several well-documented examples of successful systemscan rightly be referred to as KDD applicationsand have been deployed in operational useon large-scale real-world problems in scienceand in business.

In science, one of the primary application

There is anurgent need

for a new generation ofcomputation-

al theoriesand tools to

assist humans inextracting

useful information(knowledge)

from therapidly

growing volumes of

digital data.

Articles

38 AI MAGAZINE

tive applications (Manago and Auriol 1996).Telecommunications: The telecommuni-

cations alarm-sequence analyzer (TASA) wasbuilt in cooperation with a manufacturer oftelecommunications equipment and threetelephone networks (Mannila, Toivonen, andVerkamo 1995). The system uses a novelframework for locating frequently occurringalarm episodes from the alarm stream andpresenting them as rules. Large sets of discov-ered rules can be explored with flexible infor-mation-retrieval tools supporting interactivityand iteration. In this way, TASA offers pruning,grouping, and ordering tools to refine the re-sults of a basic brute-force search for rules.

Data cleaning: The MERGE-PURGE systemwas applied to the identification of duplicatewelfare claims (Hernandez and Stolfo 1995).It was used successfully on data from the Wel-fare Department of the State of Washington.

In other areas, a well-publicized system isIBM’s ADVANCED SCOUT, a specialized data-min-ing system that helps National Basketball As-sociation (NBA) coaches organize and inter-pret data from NBA games (U.S. News 1995).ADVANCED SCOUT was used by several of theNBA teams in 1996, including the Seattle Su-personics, which reached the NBA finals.

Finally, a novel and increasingly importanttype of discovery is one based on the use of in-telligent agents to navigate through an infor-mation-rich environment. Although the ideaof active triggers has long been analyzed in thedatabase field, really successful applications ofthis idea appeared only with the advent of theInternet. These systems ask the user to specifya profile of interest and search for related in-formation among a wide variety of public-do-main and proprietary sources. For example,FIREFLY is a personal music-recommendationagent: It asks a user his/her opinion of severalmusic pieces and then suggests other musicthat the user might like (<http://www.ffly.com/>). CRAYON (http://crayon.net/>)allows users to create their own free newspaper(supported by ads); NEWSHOUND (<http://www.sjmercury.com/hound/>) from the San JoseMercury News and FARCAST (<http://www.far-cast.com/> automatically search informationfrom a wide variety of sources, includingnewspapers and wire services, and e-mail rele-vant documents directly to the user.

These are just a few of the numerous suchsystems that use KDD techniques to automat-ically produce useful information from largemasses of raw data. See Piatetsky-Shapiro etal. (1996) for an overview of issues in devel-oping industrial KDD applications.

Data Mining and KDDHistorically, the notion of finding useful pat-terns in data has been given a variety ofnames, including data mining, knowledge ex-traction, information discovery, informationharvesting, data archaeology, and data patternprocessing. The term data mining has mostlybeen used by statisticians, data analysts, andthe management information systems (MIS)communities. It has also gained popularity inthe database field. The phrase knowledge dis-covery in databases was coined at the first KDDworkshop in 1989 (Piatetsky-Shapiro 1991) toemphasize that knowledge is the end productof a data-driven discovery. It has been popular-ized in the AI and machine-learning fields.

In our view, KDD refers to the overall pro-cess of discovering useful knowledge from da-ta, and data mining refers to a particular stepin this process. Data mining is the applicationof specific algorithms for extracting patternsfrom data. The distinction between the KDDprocess and the data-mining step (within theprocess) is a central point of this article. Theadditional steps in the KDD process, such asdata preparation, data selection, data cleaning,incorporation of appropriate prior knowledge,and proper interpretation of the results ofmining, are essential to ensure that usefulknowledge is derived from the data. Blind ap-plication of data-mining methods (rightly crit-icized as data dredging in the statistical litera-ture) can be a dangerous activity, easilyleading to the discovery of meaningless andinvalid patterns.

The Interdisciplinary Nature of KDDKDD has evolved, and continues to evolve,from the intersection of research fields such asmachine learning, pattern recognition,databases, statistics, AI, knowledge acquisitionfor expert systems, data visualization, andhigh-performance computing. The unifyinggoal is extracting high-level knowledge fromlow-level data in the context of large data sets.

The data-mining component of KDD cur-rently relies heavily on known techniquesfrom machine learning, pattern recognition,and statistics to find patterns from data in thedata-mining step of the KDD process. A natu-ral question is, How is KDD different from pat-tern recognition or machine learning (and re-lated fields)? The answer is that these fieldsprovide some of the data-mining methodsthat are used in the data-mining step of theKDD process. KDD focuses on the overall pro-cess of knowledge discovery from data, includ-ing how the data are stored and accessed, howalgorithms can be scaled to massive data sets

The basicproblem addressed bythe KDD process is one of mapping low-level data intoother formsthat might bemore compact,more abstract, or more useful.

Articles

FALL 1996 39

A driving force behind KDD is the databasefield (the second D in KDD). Indeed, theproblem of effective data manipulation whendata cannot fit in the main memory is of fun-damental importance to KDD. Database tech-niques for gaining efficient data access,grouping and ordering operations when ac-cessing data, and optimizing queries consti-tute the basics for scaling algorithms to largerdata sets. Most data-mining algorithms fromstatistics, pattern recognition, and machinelearning assume data are in the main memo-ry and pay no attention to how the algorithmbreaks down if only limited views of the dataare possible.

A related field evolving from databases isdata warehousing, which refers to the popularbusiness trend of collecting and cleaningtransactional data to make them available foronline analysis and decision support. Datawarehousing helps set the stage for KDD intwo important ways: (1) data cleaning and (2)data access.

Data cleaning: As organizations are forcedto think about a unified logical view of thewide variety of data and databases they pos-sess, they have to address the issues of map-ping data to a single naming convention,uniformly representing and handling missingdata, and handling noise and errors whenpossible.

Data access: Uniform and well-definedmethods must be created for accessing the da-ta and providing access paths to data thatwere historically difficult to get to (for exam-ple, stored offline).

Once organizations and individuals havesolved the problem of how to store and ac-cess their data, the natural next step is thequestion, What else do we do with all the da-ta? This is where opportunities for KDD natu-rally arise.

A popular approach for analysis of datawarehouses is called online analytical processing(OLAP), named for a set of principles pro-posed by Codd (1993). OLAP tools focus onproviding multidimensional data analysis,which is superior to SQL in computing sum-maries and breakdowns along many dimen-sions. OLAP tools are targeted toward simpli-fying and supporting interactive data analysis,but the goal of KDD tools is to automate asmuch of the process as possible. Thus, KDD isa step beyond what is currently supported bymost standard database systems.

Basic DefinitionsKDD is the nontrivial process of identifyingvalid, novel, potentially useful, and ultimate-

and still run efficiently, how results can be in-terpreted and visualized, and how the overallman-machine interaction can usefully bemodeled and supported. The KDD processcan be viewed as a multidisciplinary activitythat encompasses techniques beyond thescope of any one particular discipline such asmachine learning. In this context, there areclear opportunities for other fields of AI (be-sides machine learning) to contribute toKDD. KDD places a special emphasis on find-ing understandable patterns that can be inter-preted as useful or interesting knowledge.Thus, for example, neural networks, althougha powerful modeling tool, are relativelydifficult to understand compared to decisiontrees. KDD also emphasizes scaling and ro-bustness properties of modeling algorithmsfor large noisy data sets.

Related AI research fields include machinediscovery, which targets the discovery of em-pirical laws from observation and experimen-tation (Shrager and Langley 1990) (see Kloes-gen and Zytkow [1996] for a glossary of termscommon to KDD and machine discovery),and causal modeling for the inference ofcausal models from data (Spirtes, Glymour,and Scheines 1993). Statistics in particularhas much in common with KDD (see Elderand Pregibon [1996] and Glymour et al.[1996] for a more detailed discussion of thissynergy). Knowledge discovery from data isfundamentally a statistical endeavor. Statisticsprovides a language and framework for quan-tifying the uncertainty that results when onetries to infer general patterns from a particu-lar sample of an overall population. As men-tioned earlier, the term data mining has hadnegative connotations in statistics since the1960s when computer-based data analysistechniques were first introduced. The concernarose because if one searches long enough inany data set (even randomly generated data),one can find patterns that appear to be statis-tically significant but, in fact, are not. Clearly,this issue is of fundamental importance toKDD. Substantial progress has been made inrecent years in understanding such issues instatistics. Much of this work is of direct rele-vance to KDD. Thus, data mining is a legiti-mate activity as long as one understands howto do it correctly; data mining carried outpoorly (without regard to the statistical as-pects of the problem) is to be avoided. KDDcan also be viewed as encompassing a broaderview of modeling than statistics. KDD aims toprovide tools to automate (to the degree pos-sible) the entire process of data analysis andthe statistician’s “art” of hypothesis selection.

Data miningis a step in

the KDD process that

consists of ap-plying data

analysis anddiscovery al-

gorithms thatproduce a par-

ticular enu-meration of

patterns (or models)

over the data.

Articles

40 AI MAGAZINE

ly understandable patterns in data (Fayyad,Piatetsky-Shapiro, and Smyth 1996).

Here, data are a set of facts (for example,cases in a database), and pattern is an expres-sion in some language describing a subset ofthe data or a model applicable to the subset.Hence, in our usage here, extracting a patternalso designates fitting a model to data; find-ing structure from data; or, in general, mak-ing any high-level description of a set of data.The term process implies that KDD comprisesmany steps, which involve data preparation,search for patterns, knowledge evaluation,and refinement, all repeated in multiple itera-tions. By nontrivial, we mean that somesearch or inference is involved; that is, it isnot a straightforward computation ofpredefined quantities like computing the av-erage value of a set of numbers.

The discovered patterns should be valid onnew data with some degree of certainty. Wealso want patterns to be novel (at least to thesystem and preferably to the user) and poten-tially useful, that is, lead to some benefit tothe user or task. Finally, the patterns shouldbe understandable, if not immediately thenafter some postprocessing.

The previous discussion implies that we candefine quantitative measures for evaluatingextracted patterns. In many cases, it is possi-ble to define measures of certainty (for exam-ple, estimated prediction accuracy on new

data) or utility (for example, gain, perhaps indollars saved because of better predictions orspeedup in response time of a system). No-tions such as novelty and understandabilityare much more subjective. In certain contexts,understandability can be estimated by sim-plicity (for example, the number of bits to de-scribe a pattern). An important notion, calledinterestingness (for example, see Silberschatzand Tuzhilin [1995] and Piatetsky-Shapiro andMatheus [1994]), is usually taken as an overallmeasure of pattern value, combining validity,novelty, usefulness, and simplicity. Interest-ingness functions can be defined explicitly orcan be manifested implicitly through an or-dering placed by the KDD system on the dis-covered patterns or models.

Given these notions, we can consider apattern to be knowledge if it exceeds some in-terestingness threshold, which is by nomeans an attempt to define knowledge in thephilosophical or even the popular view. As amatter of fact, knowledge in this definition ispurely user oriented and domain specific andis determined by whatever functions andthresholds the user chooses.

Data mining is a step in the KDD processthat consists of applying data analysis anddiscovery algorithms that, under acceptablecomputational efficiency limitations, pro-duce a particular enumeration of patterns (ormodels) over the data. Note that the space of

Articles

FALL 1996 41

Data

Transformed�Data

Patterns

Preprocessing

Data Mining

Interpretation / �Evaluation

Transformation

Selection

--- --- ------ --- ------ --- ---

Knowledge

Preprocessed Data

Target Date

Figure 1. An Overview of the Steps That Compose the KDD Process.

methods, the effective number of variablesunder consideration can be reduced, or in-variant representations for the data can befound.

Fifth is matching the goals of the KDD pro-cess (step 1) to a particular data-miningmethod. For example, summarization, clas-sification, regression, clustering, and so on,are described later as well as in Fayyad, Piatet-sky-Shapiro, and Smyth (1996).

Sixth is exploratory analysis and modeland hypothesis selection: choosing the data-mining algorithm(s) and selecting method(s)to be used for searching for data patterns.This process includes deciding which modelsand parameters might be appropriate (for ex-ample, models of categorical data are differ-ent than models of vectors over the reals) andmatching a particular data-mining methodwith the overall criteria of the KDD process(for example, the end user might be more in-terested in understanding the model than itspredictive capabilities).

Seventh is data mining: searching for pat-terns of interest in a particular representa-tional form or a set of such representations,including classification rules or trees, regres-sion, and clustering. The user can significant-ly aid the data-mining method by correctlyperforming the preceding steps.

Eighth is interpreting mined patterns, pos-sibly returning to any of steps 1 through 7 forfurther iteration. This step can also involvevisualization of the extracted patterns andmodels or visualization of the data given theextracted models.

Ninth is acting on the discovered knowl-edge: using the knowledge directly, incorpo-rating the knowledge into another system forfurther action, or simply documenting it andreporting it to interested parties. This processalso includes checking for and resolving po-tential conflicts with previously believed (orextracted) knowledge.

The KDD process can involve significantiteration and can contain loops betweenany two steps. The basic flow of steps (al-though not the potential multitude of itera-tions and loops) is illustrated in figure 1.Most previous work on KDD has focused onstep 7, the data mining. However, the othersteps are as important (and probably moreso) for the successful application of KDD inpractice. Having defined the basic notionsand introduced the KDD process, we nowfocus on the data-mining component,which has, by far, received the most atten-tion in the literature.

patterns is often infinite, and the enumera-tion of patterns involves some form ofsearch in this space. Practical computationalconstraints place severe limits on the sub-space that can be explored by a data-miningalgorithm.

The KDD process involves using thedatabase along with any required selection,preprocessing, subsampling, and transforma-tions of it; applying data-mining methods(algorithms) to enumerate patterns from it;and evaluating the products of data miningto identify the subset of the enumerated pat-terns deemed knowledge. The data-miningcomponent of the KDD process is concernedwith the algorithmic means by which pat-terns are extracted and enumerated from da-ta. The overall KDD process (figure 1) in-cludes the evaluation and possibleinterpretation of the mined patterns to de-termine which patterns can be considerednew knowledge. The KDD process also in-cludes all the additional steps described inthe next section.

The notion of an overall user-driven pro-cess is not unique to KDD: analogous propos-als have been put forward both in statistics(Hand 1994) and in machine learning (Brod-ley and Smyth 1996).

The KDD ProcessThe KDD process is interactive and iterative,involving numerous steps with many deci-sions made by the user. Brachman and Anand(1996) give a practical view of the KDD pro-cess, emphasizing the interactive nature ofthe process. Here, we broadly outline some ofits basic steps:

First is developing an understanding of theapplication domain and the relevant priorknowledge and identifying the goal of theKDD process from the customer’s viewpoint.

Second is creating a target data set: select-ing a data set, or focusing on a subset of vari-ables or data samples, on which discovery isto be performed.

Third is data cleaning and preprocessing.Basic operations include removing noise ifappropriate, collecting the necessary informa-tion to model or account for noise, decidingon strategies for handling missing data fields,and accounting for time-sequence informa-tion and known changes.

Fourth is data reduction and projection:finding useful features to represent the datadepending on the goal of the task. With di-mensionality reduction or transformation

Articles

42 AI MAGAZINE

The Data-Mining Step of the KDD Process

The data-mining component of the KDD pro-cess often involves repeated iterative applica-tion of particular data-mining methods. Thissection presents an overview of the primarygoals of data mining, a description of themethods used to address these goals, and abrief description of the data-mining algo-rithms that incorporate these methods.

The knowledge discovery goals are definedby the intended use of the system. We candistinguish two types of goals: (1) verificationand (2) discovery. With verification, the sys-tem is limited to verifying the user’s hypothe-sis. With discovery, the system autonomouslyfinds new patterns. We further subdivide thediscovery goal into prediction, where the sys-tem finds patterns for predicting the futurebehavior of some entities, and description,where the system finds patterns for presenta-tion to a user in a human-understandableform. In this article, we are primarily con-cerned with discovery-oriented data mining.

Data mining involves fitting models to, ordetermining patterns from, observed data.The fitted models play the role of inferredknowledge: Whether the models reflect usefulor interesting knowledge is part of the over-all, interactive KDD process where subjectivehuman judgment is typically required. Twoprimary mathematical formalisms are used inmodel fitting: (1) statistical and (2) logical.The statistical approach allows for nondeter-ministic effects in the model, whereas a logi-cal model is purely deterministic. We focusprimarily on the statistical approach to datamining, which tends to be the most widelyused basis for practical data-mining applica-tions given the typical presence of uncertain-ty in real-world data-generating processes.

Most data-mining methods are based ontried and tested techniques from machinelearning, pattern recognition, and statistics:classification, clustering, regression, and soon. The array of different algorithms undereach of these headings can often be bewilder-ing to both the novice and the experienceddata analyst. It should be emphasized that ofthe many data-mining methods advertised inthe literature, there are really only a few fun-damental techniques. The actual underlyingmodel representation being used by a particu-lar method typically comes from a composi-tion of a small number of well-known op-tions: polynomials, splines, kernel and basisfunctions, threshold-Boolean functions, andso on. Thus, algorithms tend to differ primar-

ily in the goodness-of-fit criterion used toevaluate model fit or in the search methodused to find a good fit.

In our brief overview of data-mining meth-ods, we try in particular to convey the notionthat most (if not all) methods can be viewedas extensions or hybrids of a few basic tech-niques and principles. We first discuss the pri-mary methods of data mining and then showthat the data- mining methods can be viewedas consisting of three primary algorithmiccomponents: (1) model representation, (2)model evaluation, and (3) search. In the dis-cussion of KDD and data-mining methods,we use a simple example to make some of thenotions more concrete. Figure 2 shows a sim-ple two-dimensional artificial data set consist-ing of 23 cases. Each point on the graph rep-resents a person who has been given a loanby a particular bank at some time in the past.The horizontal axis represents the income ofthe person; the vertical axis represents the to-tal personal debt of the person (mortgage, carpayments, and so on). The data have beenclassified into two classes: (1) the x’s repre-sent persons who have defaulted on theirloans and (2) the o’s represent persons whoseloans are in good status with the bank. Thus,this simple artificial data set could represent ahistorical data set that can contain usefulknowledge from the point of view of thebank making the loans. Note that in actualKDD applications, there are typically manymore dimensions (as many as several hun-dreds) and many more data points (manythousands or even millions).

Articles

FALL 1996 43

xx

x

x

o

o

o

o

Income

Debt

ox

o

o

o

o

o ox

x

x

x

x

o

o

Figure 2. A Simple Data Set with Two Classes Used for Illustrative Purposes.

The purpose here is to illustrate basic ideason a small problem in two-dimensionalspace.

Data-Mining MethodsThe two high-level primary goals of data min-ing in practice tend to be prediction and de-scription. As stated earlier, prediction in-volves using some variables or fields in thedatabase to predict unknown or future valuesof other variables of interest, and descriptionfocuses on finding human-interpretable pat-terns describing the data. Although theboundaries between prediction and descrip-tion are not sharp (some of the predictivemodels can be descriptive, to the degree thatthey are understandable, and vice versa), thedistinction is useful for understanding theoverall discovery goal. The relative impor-tance of prediction and description for partic-ular data-mining applications can vary con-siderably. The goals of prediction anddescription can be achieved using a variety ofparticular data-mining methods.

Classification is learning a function thatmaps (classifies) a data item into one of sever-al predefined classes (Weiss and Kulikowski1991; Hand 1981). Examples of classificationmethods used as part of knowledge discoveryapplications include the classifying of trendsin financial markets (Apte and Hong 1996)and the automated identification of objects ofinterest in large image databases (Fayyad,Djorgovski, and Weir 1996). Figure 3 shows asimple partitioning of the loan data into twoclass regions; note that it is not possible toseparate the classes perfectly using a lineardecision boundary. The bank might want touse the classification regions to automaticallydecide whether future loan applicants will begiven a loan or not.

Regression is learning a function that mapsa data item to a real-valued prediction vari-able. Regression applications are many, forexample, predicting the amount of biomasspresent in a forest given remotely sensed mi-crowave measurements, estimating the proba-bility that a patient will survive given the re-sults of a set of diagnostic tests, predictingconsumer demand for a new product as afunction of advertising expenditure, and pre-dicting time series where the input variablescan be time-lagged versions of the predictionvariable. Figure 4 shows the result of simplelinear regression where total debt is fitted as alinear function of income: The fit is poor be-cause only a weak correlation exists betweenthe two variables.

Clustering is a common descriptive task

Articles

44 AI MAGAZINE

Figure 3. A Simple Linear Classification Boundary for the Loan Data Set.The shaped region denotes class no loan.

xx

x

x

o

o

o

o

Income

Debt

ox

o

o

o

o

o ox

x

x

x

x

oNo Loan

Loan

o

Figure 4. A Simple Linear Regression for the Loan Data Set.

xx

x

x

o

o

o

o

Income

Debt

ox

o

o

o

o

o ox

x

x

x

x

o

o

Regression Line

where one seeks to identify a finite set of cat-egories or clusters to describe the data (Jainand Dubes 1988; Titterington, Smith, andMakov 1985). The categories can be mutuallyexclusive and exhaustive or consist of a richerrepresentation, such as hierarchical or over-lapping categories. Examples of clustering ap-plications in a knowledge discovery contextinclude discovering homogeneous subpopula-tions for consumers in marketing databasesand identifying subcategories of spectra frominfrared sky measurements (Cheeseman andStutz 1996). Figure 5 shows a possible cluster-ing of the loan data set into three clusters;note that the clusters overlap, allowing datapoints to belong to more than one cluster.The original class labels (denoted by x’s ando’s in the previous figures) have been replacedby a + to indicate that the class membershipis no longer assumed known. Closely relatedto clustering is the task of probability densityestimation, which consists of techniques forestimating from data the joint multivariateprobability density function of all the vari-ables or fields in the database (Silverman1986).

Summarization involves methods for find-ing a compact description for a subset of da-ta. A simple example would be tabulating themean and standard deviations for all fields.More sophisticated methods involve thederivation of summary rules (Agrawal et al.1996), multivariate visualization techniques,and the discovery of functional relationshipsbetween variables (Zembowicz and Zytkow1996). Summarization techniques are oftenapplied to interactive exploratory data analy-sis and automated report generation.

Dependency modeling consists of finding amodel that describes significant dependenciesbetween variables. Dependency models existat two levels: (1) the structural level of themodel specifies (often in graphic form) whichvariables are locally dependent on each otherand (2) the quantitative level of the modelspecifies the strengths of the dependenciesusing some numeric scale. For example, prob-abilistic dependency networks use condition-al independence to specify the structural as-pect of the model and probabilities orcorrelations to specify the strengths of the de-pendencies (Glymour et al. 1987; Heckerman1996). Probabilistic dependency networks areincreasingly finding applications in areas asdiverse as the development of probabilisticmedical expert systems from databases, infor-mation retrieval, and modeling of the humangenome.

Change and deviation detection focuses on

discovering the most significant changes inthe data from previously measured or norma-tive values (Berndt and Clifford 1996; Guyon,Matic, and Vapnik 1996; Kloesgen 1996;Matheus, Piatetsky-Shapiro, and McNeill1996; Basseville and Nikiforov 1993).

The Components of Data-Mining AlgorithmsThe next step is to construct specific algo-rithms to implement the general methods weoutlined. One can identify three primarycomponents in any data-mining algorithm:(1) model representation, (2) model evalua-tion, and (3) search.

This reductionist view is not necessarilycomplete or fully encompassing; rather, it is aconvenient way to express the key conceptsof data-mining algorithms in a relativelyunified and compact manner. Cheeseman(1990) outlines a similar structure.

Model representation is the language used todescribe discoverable patterns. If the repre-sentation is too limited, then no amount oftraining time or examples can produce an ac-curate model for the data. It is important thata data analyst fully comprehend the represen-tational assumptions that might be inherentin a particular method. It is equally impor-tant that an algorithm designer clearly statewhich representational assumptions are beingmade by a particular algorithm. Note that in-creased representational power for models in-creases the danger of overfitting the trainingdata, resulting in reduced prediction accuracyon unseen data.

Model-evaluation criteria are quantitative

Articles

FALL 1996 45

++

+

+

+

+

+

+

Income

Debt

++

+

+

+

+

++

+

+

+

+

+

+

+

Cluster 2

Cluster 3

Cluster 1

Figure 5. A Simple Clustering of the Loan Data Set into Three Clusters.Note that original labels are replaced by a +.

Decision Trees and RulesDecision trees and rules that use univariatesplits have a simple representational form,making the inferred model relatively easy forthe user to comprehend. However, the restric-tion to a particular tree or rule representationcan significantly restrict the functional form(and, thus, the approximation power) of themodel. For example, figure 6 illustrates the ef-fect of a threshold split applied to the incomevariable for a loan data set: It is clear that us-ing such simple threshold splits (parallel tothe feature axes) severely limits the type ofclassification boundaries that can be induced.If one enlarges the model space to allow moregeneral expressions (such as multivariate hy-perplanes at arbitrary angles), then the modelis more powerful for prediction but can bemuch more difficult to comprehend. A largenumber of decision tree and rule-inductionalgorithms are described in the machine-learning and applied statistics literature(Quinlan 1992; Breiman et al. 1984).

To a large extent, they depend on likeli-hood-based model-evaluation methods, withvarying degrees of sophistication in terms ofpenalizing model complexity. Greedy searchmethods, which involve growing and prun-ing rule and tree structures, are typically usedto explore the superexponential space of pos-sible models. Trees and rules are primarilyused for predictive modeling, both for clas-sification (Apte and Hong 1996; Fayyad, Djor-govski, and Weir 1996) and regression, al-though they can also be applied to summarydescriptive modeling (Agrawal et al. 1996).

Nonlinear Regression and Classification MethodsThese methods consist of a family of tech-niques for prediction that fit linear and non-linear combinations of basis functions (sig-moids, splines, polynomials) to combinationsof the input variables. Examples include feed-forward neural networks, adaptive splinemethods, and projection pursuit regression(see Elder and Pregibon [1996], Cheng andTitterington [1994], and Friedman [1989] formore detailed discussions). Consider neuralnetworks, for example. Figure 7 illustrates thetype of nonlinear decision boundary that aneural network might find for the loan dataset. In terms of model evaluation, althoughnetworks of the appropriate size can univer-sally approximate any smooth function toany desired degree of accuracy, relatively littleis known about the representation propertiesof fixed-size networks estimated from finitedata sets. Also, the standard squared error and

statements (or fit functions) of how well a par-ticular pattern (a model and its parameters)meets the goals of the KDD process. For ex-ample, predictive models are often judged bythe empirical prediction accuracy on sometest set. Descriptive models can be evaluatedalong the dimensions of predictive accuracy,novelty, utility, and understandability of thefitted model.

Search method consists of two components:(1) parameter search and (2) model search.Once the model representation (or family ofrepresentations) and the model-evaluationcriteria are fixed, then the data-mining prob-lem has been reduced to purely an optimiza-tion task: Find the parameters and modelsfrom the selected family that optimize theevaluation criteria. In parameter search, thealgorithm must search for the parametersthat optimize the model-evaluation criteriagiven observed data and a fixed model repre-sentation. Model search occurs as a loop overthe parameter-search method: The model rep-resentation is changed so that a family ofmodels is considered.

Some Data-Mining MethodsA wide variety of data-mining methods exist,but here, we only focus on a subset of popu-lar techniques. Each method is discussed inthe context of model representation, modelevaluation, and search.

Articles

46 AI MAGAZINE

xx

x

x

o

o

o

o

Income

Debt

ox

o

o

o

o

o ox

x

x

x

x

oNo Loan

Loan

o

t

Figure 6. Using a Single Threshold on the Income Variable to Try to Classify the Loan Data Set.

cross-entropy loss functions used to trainneural networks can be viewed as log-likeli-hood functions for regression andclassification, respectively (Ripley 1994; Ge-man, Bienenstock, and Doursat 1992). Backpropagation is a parameter-search methodthat performs gradient descent in parameter(weight) space to find a local maximum ofthe likelihood function starting from randominitial conditions. Nonlinear regression meth-ods, although powerful in representationalpower, can be difficult to interpret.

For example, although the classificationboundaries of figure 7 might be more accu-rate than the simple threshold boundary offigure 6, the threshold boundary has the ad-vantage that the model can be expressed, tosome degree of certainty, as a simple rule ofthe form “if income is greater than threshold,then loan will have good status.”

Example-Based MethodsThe representation is simple: Use representa-tive examples from the database to approxi-mate a model; that is, predictions on new ex-amples are derived from the properties ofsimilar examples in the model whose predic-tion is known. Techniques include nearest-neighbor classification and regression algo-rithms (Dasarathy 1991) and case-basedreasoning systems (Kolodner 1993). Figure 8illustrates the use of a nearest-neighbor clas-sifier for the loan data set: The class at anynew point in the two-dimensional space isthe same as the class of the closest point inthe original training data set.

A potential disadvantage of example-basedmethods (compared with tree-based methods)is that a well-defined distance metric for eval-uating the distance between data points is re-quired. For the loan data in figure 8, thiswould not be a problem because income anddebt are measured in the same units. Howev-er, if one wished to include variables such asthe duration of the loan, sex, and profession,then it would require more effort to define asensible metric between the variables. Modelevaluation is typically based on cross-valida-tion estimates (Weiss and Kulikowski 1991) ofa prediction error: Parameters of the model tobe estimated can include the number ofneighbors to use for prediction and the dis-tance metric itself. Like nonlinear regressionmethods, example-based methods are oftenasymptotically powerful in terms of approxi-mation properties but, conversely, can bedifficult to interpret because the model is im-plicit in the data and not explicitly formulat-ed. Related techniques include kernel-density

Articles

FALL 1996 47

xx

x

x

o

o

o

o

Income

Debt

ox

o

o

o

o

o ox

x

x

x

x

oNo Loan

Loan

o

Figure 7. An Example of Classification Boundaries Learned by a NonlinearClassifier (Such as a Neural Network) for the Loan Data Set.

xx

x

x

o

o

o

o

Income

Debt

ox

o

o

o

o

o ox

x

x

x

x

oNo Loan

Loan

o

Figure 8. Classification Boundaries for a Nearest-Neighbor Classifier for the Loan Data Set.

evitably limited in scope; many data-miningtechniques, particularly specialized methodsfor particular types of data and domains, werenot mentioned specifically. We believe thegeneral discussion on data-mining tasks andcomponents has general relevance to a vari-ety of methods. For example, consider time-series prediction, which traditionally hasbeen cast as a predictive regression task (au-toregressive models, and so on). Recently,more general models have been developed fortime-series applications, such as nonlinear ba-sis functions, example-based models, and ker-nel methods. Furthermore, there has beensignificant interest in descriptive graphic andlocal data modeling of time series rather thanpurely predictive modeling (Weigend andGershenfeld 1993). Thus, although differentalgorithms and applications might appear dif-ferent on the surface, it is not uncommon tofind that they share many common compo-nents. Understanding data mining and modelinduction at this component level clarifiesthe behavior of any data-mining algorithmand makes it easier for the user to understandits overall contribution and applicability tothe KDD process.

An important point is that each techniquetypically suits some problems better thanothers. For example, decision tree classifierscan be useful for finding structure in high-di-mensional spaces and in problems withmixed continuous and categorical data (be-cause tree methods do not require distancemetrics). However, classification trees mightnot be suitable for problems where the truedecision boundaries between classes are de-scribed by a second-order polynomial (for ex-ample). Thus, there is no universal data-min-ing method, and choosing a particularalgorithm for a particular application is some-thing of an art. In practice, a large portion ofthe application effort can go into properlyformulating the problem (asking the rightquestion) rather than into optimizing the al-gorithmic details of a particular data-miningmethod (Langley and Simon 1995; Hand1994).

Because our discussion and overview of da-ta-mining methods has been brief, we wantto make two important points clear:

First, our overview of automated search fo-cused mainly on automated methods for ex-tracting patterns or models from data. Al-though this approach is consistent with thedefinition we gave earlier, it does not neces-sarily represent what other communitiesmight refer to as data mining. For example,some use the term to designate any manual

estimation (Silverman 1986) and mixturemodeling (Titterington, Smith, and Makov1985).

Probabilistic Graphic Dependency ModelsGraphic models specify probabilistic depen-dencies using a graph structure (Whittaker1990; Pearl 1988). In its simplest form, themodel specifies which variables are directly de-pendent on each other. Typically, these mod-els are used with categorical or discrete-valuedvariables, but extensions to special cases, suchas Gaussian densities, for real-valued variablesare also possible. Within the AI and statisticalcommunities, these models were initially de-veloped within the framework of probabilisticexpert systems; the structure of the model andthe parameters (the conditional probabilitiesattached to the links of the graph) were elicit-ed from experts. Recently, there has been sig-nificant work in both the AI and statisticalcommunities on methods whereby both thestructure and the parameters of graphic mod-els can be learned directly from databases(Buntine 1996; Heckerman 1996). Model-eval-uation criteria are typically Bayesian in form,and parameter estimation can be a mixture ofclosed-form estimates and iterative methodsdepending on whether a variable is directlyobserved or hidden. Model search can consistof greedy hill-climbing methods over variousgraph structures. Prior knowledge, such as apartial ordering of the variables based oncausal relations, can be useful in terms of re-ducing the model search space. Although stillprimarily in the research phase, graphic modelinduction methods are of particular interest toKDD because the graphic form of the modellends itself easily to human interpretation.

Relational Learning ModelsAlthough decision trees and rules have a repre-sentation restricted to propositional logic, rela-tional learning (also known as inductive logicprogramming) uses the more flexible patternlanguage of first-order logic. A relational learn-er can easily find formulas such as X = Y. Mostresearch to date on model-evaluation methodsfor relational learning is logical in nature. Theextra representational power of relationalmodels comes at the price of significant com-putational demands in terms of search. SeeDzeroski (1996) for a more detailed discussion.

DiscussionGiven the broad spectrum of data-miningmethods and algorithms, our overview is in-

Understand-ing data

mining andmodel

induction atthis

componentlevel clarifiesthe behavior

of any data-mining

algorithmand makes iteasier for the

user to understand its

overall contribution

and applicability

to the KDD

process.

Articles

48 AI MAGAZINE

search of the data or search assisted by queriesto a database management system or to referto humans visualizing patterns in data. Inother communities, it is used to refer to theautomated correlation of data from transac-tions or the automated generation of transac-tion reports. We choose to focus only onmethods that contain certain degrees ofsearch autonomy.

Second, beware the hype: The state of theart in automated methods in data mining isstill in a fairly early stage of development.There are no established criteria for decidingwhich methods to use in which circum-stances, and many of the approaches arebased on crude heuristic approximations toavoid the expensive search required to findoptimal, or even good, solutions. Hence, thereader should be careful when confrontedwith overstated claims about the great abilityof a system to mine useful information fromlarge (or even small) databases.

Application IssuesFor a survey of KDD applications as well asdetailed examples, see Piatetsky-Shapiro et al.(1996) for industrial applications and Fayyad,Haussler, and Stolorz (1996) for applicationsin science data analysis. Here, we examinecriteria for selecting potential applications,which can be divided into practical and tech-nical categories. The practical criteria for KDDprojects are similar to those for other applica-tions of advanced technology and include thepotential impact of an application, the ab-sence of simpler alternative solutions, andstrong organizational support for using tech-nology. For applications dealing with person-al data, one should also consider the privacyand legal issues (Piatetsky-Shapiro 1995).

The technical criteria include considera-tions such as the availability of sufficient data(cases). In general, the more fields there areand the more complex the patterns beingsought, the more data are needed. However,strong prior knowledge (see discussion later)can reduce the number of needed cases sig-nificantly. Another consideration is the rele-vance of attributes. It is important to have da-ta attributes that are relevant to the discoverytask; no amount of data will allow predictionbased on attributes that do not capture therequired information. Furthermore, low noiselevels (few data errors) are another considera-tion. High amounts of noise make it hard toidentify patterns unless a large number of cas-es can mitigate random noise and help clarifythe aggregate patterns. Changing and time-

oriented data, although making the applica-tion development more difficult, make it po-tentially much more useful because it is easierto retrain a system than a human. Finally,and perhaps one of the most important con-siderations, is prior knowledge. It is useful toknow something about the domain —whatare the important fields, what are the likelyrelationships, what is the user utility func-tion, what patterns are already known, and soon.

Research and Application ChallengesWe outline some of the current primary re-search and application challenges for KDD.This list is by no means exhaustive and is in-tended to give the reader a feel for the typesof problem that KDD practitioners wrestlewith.

Larger databases: Databases with hun-dreds of fields and tables and millions ofrecords and of a multigigabyte size are com-monplace, and terabyte (1012 bytes) databasesare beginning to appear. Methods for dealingwith large data volumes include moreefficient algorithms (Agrawal et al. 1996),sampling, approximation, and massively par-allel processing (Holsheimer et al. 1996).

High dimensionality: Not only is there of-ten a large number of records in the database,but there can also be a large number of fields(attributes, variables); so, the dimensionalityof the problem is high. A high-dimensionaldata set creates problems in terms of increas-ing the size of the search space for model in-duction in a combinatorially explosive man-ner. In addition, it increases the chances thata data-mining algorithm will find spuriouspatterns that are not valid in general. Ap-proaches to this problem include methods toreduce the effective dimensionality of theproblem and the use of prior knowledge toidentify irrelevant variables.

Overfitting: When the algorithm searchesfor the best parameters for one particularmodel using a limited set of data, it can mod-el not only the general patterns in the databut also any noise specific to the data set, re-sulting in poor performance of the model ontest data. Possible solutions include cross-vali-dation, regularization, and other sophisticat-ed statistical strategies.

Assessing of statistical significance: Aproblem (related to overfitting) occurs whenthe system is searching over many possiblemodels. For example, if a system tests modelsat the 0.001 significance level, then on aver-age, with purely random data, N/1000 ofthese models will be accepted as significant.

Articles

FALL 1996 49

edge is important in all the steps of the KDDprocess. Bayesian approaches (for example,Cheeseman [1990]) use prior probabilitiesover data and distributions as one form of en-coding prior knowledge. Others employ de-ductive database capabilities to discoverknowledge that is then used to guide the da-ta-mining search (for example, Simoudis,Livezey, and Kerber [1995]).

Integration with other systems: A stand-alone discovery system might not be veryuseful. Typical integration issues include inte-gration with a database management system(for example, through a query interface), in-tegration with spreadsheets and visualizationtools, and accommodating of real-time sensorreadings. Examples of integrated KDD sys-tems are described by Simoudis, Livezey, andKerber (1995) and Stolorz, Nakamura, Mesro-biam, Muntz, Shek, Santos, Yi, Ng, Chien,Mechoso, and Farrara (1995).

Concluding Remarks: The Potential Role of AI in KDD

In addition to machine learning, other AI fiel-ds can potentially contribute significantly tovarious aspects of the KDD process. We men-tion a few examples of these areas here:

Natural language presents significant op-portunities for mining in free-form text, espe-cially for automated annotation and indexingprior to classification of text corpora. Limitedparsing capabilities can help substantially inthe task of deciding what an article refers to.Hence, the spectrum from simple natural lan-guage processing all the way to language un-derstanding can help substantially. Also, nat-ural language processing can contributesignificantly as an effective interface for stat-ing hints to mining algorithms and visualiz-ing and explaining knowledge derived by aKDD system.

Planning considers a complicated dataanalysis process. It involves conducting com-plicated data-access and data-transformationoperations; applying preprocessing routines;and, in some cases, paying attention to re-source and data-access constraints. Typically,data processing steps are expressed in terms ofdesired postconditions and preconditions forthe application of certain routines, whichlends itself easily to representation as a plan-ning problem. In addition, planning abilitycan play an important role in automatedagents (see next item) to collect data samplesor conduct a search to obtain needed data sets.

Intelligent agents can be fired off to col-lect necessary information from a variety of

This point is frequently missed by many ini-tial attempts at KDD. One way to deal withthis problem is to use methods that adjustthe test statistic as a function of the search,for example, Bonferroni adjustments for inde-pendent tests or randomization testing.

Changing data and knowledge: Rapidlychanging (nonstationary) data can make pre-viously discovered patterns invalid. In addi-tion, the variables measured in a given appli-cation database can be modified, deleted, oraugmented with new measurements overtime. Possible solutions include incrementalmethods for updating the patterns and treat-ing change as an opportunity for discoveryby using it to cue the search for patterns ofchange only (Matheus, Piatetsky-Shapiro, andMcNeill 1996). See also Agrawal and Psaila(1995) and Mannila, Toivonen, and Verkamo(1995).

Missing and noisy data: This problem isespecially acute in business databases. U.S.census data reportedly have error rates asgreat as 20 percent in some fields. Importantattributes can be missing if the database wasnot designed with discovery in mind. Possiblesolutions include more sophisticated statisti-cal strategies to identify hidden variables anddependencies (Heckerman 1996; Smyth et al.1996).

Complex relationships between fields:Hierarchically structured attributes or values,relations between attributes, and more so-phisticated means for representing knowl-edge about the contents of a database will re-quire algorithms that can effectively use suchinformation. Historically, data-mining algo-rithms have been developed for simple at-tribute-value records, although new tech-niques for deriving relations betweenvariables are being developed (Dzeroski 1996;Djoko, Cook, and Holder 1995).

Understandability of patterns: In manyapplications, it is important to make the dis-coveries more understandable by humans.Possible solutions include graphic representa-tions (Buntine 1996; Heckerman 1996), rulestructuring, natural language generation, andtechniques for visualization of data andknowledge. Rule-refinement strategies (for ex-ample, Major and Mangano [1995]) can beused to address a related problem: The discov-ered knowledge might be implicitly or explic-itly redundant.

User interaction and prior knowledge:Many current KDD methods and tools are nottruly interactive and cannot easily incorpo-rate prior knowledge about a problem exceptin simple ways. The use of domain knowl-

Articles

50 AI MAGAZINE

sources. In addition, information agents canbe activated remotely over the network orcan trigger on the occurrence of a certainevent and start an analysis operation. Finally,agents can help navigate and model theWorld-Wide Web (Etzioni 1996), another areagrowing in importance.

Uncertainty in AI includes issues for man-aging uncertainty, proper inference mecha-nisms in the presence of uncertainty, and thereasoning about causality, all fundamental toKDD theory and practice. In fact, the KDD-96conference had a joint session with the UAI-96conference this year (Horvitz and Jensen 1996).

Knowledge representation includes on-tologies, new concepts for representing, stor-ing, and accessing knowledge. Also includedare schemes for representing knowledge andallowing the use of prior human knowledgeabout the underlying process by the KDDsystem.

These potential contributions of AI are buta sampling; many others, including human-computer interaction, knowledge-acquisitiontechniques, and the study of mechanisms forreasoning, have the opportunity to con-tribute to KDD.

In conclusion, we presented some defini-tions of basic notions in the KDD field. Ourprimary aim was to clarify the relation be-tween knowledge discovery and data mining.We provided an overview of the KDD processand basic data-mining methods. Given thebroad spectrum of data-mining methods andalgorithms, our overview is inevitably limit-ed in scope: There are many data-miningtechniques, particularly specialized methodsfor particular types of data and domain. Al-though various algorithms and applicationsmight appear quite different on the surface,it is not uncommon to find that they sharemany common components. Understandingdata mining and model induction at thiscomponent level clarifies the task of any da-ta-mining algorithm and makes it easier forthe user to understand its overall contribu-tion and applicability to the KDD process.

This article represents a step toward acommon framework that we hope will ulti-mately provide a unifying vision of the com-mon overall goals and methods used inKDD. We hope this will eventually lead to abetter understanding of the variety of ap-proaches in this multidisciplinary field andhow they fit together.

AcknowledgmentsWe thank Sam Uthurusamy, Ron Brachman, andKDD-96 referees for their valuable suggestionsand ideas.

Note1. Throughout this article, we use the term patternto designate a pattern found in data. We also referto models. One can think of patterns as compo-nents of models, for example, a particular rule in aclassification model or a linear component in a re-gression model.

ReferencesAgrawal, R., and Psaila, G. 1995. Active Data Min-ing. In Proceedings of the First International Con-ference on Knowledge Discovery and Data Mining(KDD-95), 3–8. Menlo Park, Calif.: American Asso-ciation for Artificial Intelligence.Agrawal, R.; Mannila, H.; Srikant, R.; Toivonen, H.;and Verkamo, I. 1996. Fast Discovery of AssociationRules. In Advances in Knowledge Discovery and DataMining, eds. U. Fayyad, G. Piatetsky-Shapiro, P.Smyth, and R. Uthurusamy, 307–328. Menlo Park,Calif.: AAAI Press.Apte, C., and Hong, S. J. 1996. Predicting EquityReturns from Securities Data with Minimal RuleGeneration. In Advances in Knowledge Discovery andData Mining, eds. U. Fayyad, G. Piatetsky-Shapiro, P.Smyth, and R. Uthurusamy, 514–560. Menlo Park,Calif.: AAAI Press.Basseville, M., and Nikiforov, I. V. 1993. Detectionof Abrupt Changes: Theory and Application. Engle-wood Cliffs, N.J.: Prentice Hall.Berndt, D., and Clifford, J. 1996. Finding Patternsin Time Series: A Dynamic Programming Approach.In Advances in Knowledge Discovery and Data Mining,eds. U. Fayyad, G. Piatetsky-Shapiro, P. Smyth, andR. Uthurusamy, 229–248. Menlo Park, Calif.: AAAIPress.Berry, J. 1994. Database Marketing. Business Week,September 5, 56–62.Brachman, R., and Anand, T. 1996. The Process ofKnowledge Discovery in Databases: A Human-Cen-tered Approach. In Advances in Knowledge Discoveryand Data Mining, 37–58, eds. U. Fayyad, G. Piatet-sky-Shapiro, P. Smyth, and R. Uthurusamy. MenloPark, Calif.: AAAI Press.Breiman, L.; Friedman, J. H.; Olshen, R. A.; andStone, C. J. 1984. Classification and Regression Trees.Belmont, Calif.: Wadsworth.Brodley, C. E., and Smyth, P. 1996. Applying Clas-sification Algorithms in Practice. Statistics and Com-puting. Forthcoming.Buntine, W. 1996. Graphical Models for Discover-ing Knowledge. In Advances in Knowledge Discoveryand Data Mining, eds. U. Fayyad, G. Piatetsky-Shapiro, P. Smyth, and R. Uthurusamy, 59–82.Menlo Park, Calif.: AAAI Press.Cheeseman, P. 1990. On Finding the Most ProbableModel. In Computational Models of Scientific Discov-ery and Theory Formation, eds. J. Shrager and P. Lan-gley, 73–95. San Francisco, Calif.: Morgan Kauf-mann.Cheeseman, P., and Stutz, J. 1996. Bayesian Clas-sification (AUTOCLASS): Theory and Results. In Ad-vances in Knowledge Discovery and Data Mining, eds.

Articles

FALL 1996 51

ering Informative Patterns and Data Cleaning. InAdvances in Knowledge Discovery and Data Mining,eds. U. Fayyad, G. Piatetsky-Shapiro, P. Smyth, andR. Uthurusamy, 181–204. Menlo Park, Calif.: AAAIPress.Hall, J.; Mani, G.; and Barr, D. 1996. ApplyingComputational Intelligence to the Investment Pro-cess. In Proceedings of CIFER-96: ComputationalIntelligence in Financial Engineering. Washington,D.C.: IEEE Computer Society.Hand, D. J. 1994. Deconstructing Statistical Ques-tions. Journal of the Royal Statistical Society A. 157(3):317–356.Hand, D. J. 1981. Discrimination and Classification.Chichester, U.K.: Wiley.Heckerman, D. 1996. Bayesian Networks for Knowl-edge Discovery. In Advances in Knowledge Discoveryand Data Mining, eds. U. Fayyad, G. Piatetsky-Shapiro, P. Smyth, and R. Uthurusamy, 273–306.Menlo Park, Calif.: AAAI Press.Hernandez, M., and Stolfo, S. 1995. The MERGE-PURGE Problem for Large Databases. In Proceedingsof the 1995 ACM-SIGMOD Conference, 127–138.New York: Association for Computing Machinery.Holsheimer, M.; Kersten, M. L.; Mannila, H.; andToivonen, H. 1996. Data Surveyor: Searching theNuggets in Parallel. In Advances in Knowledge Dis-covery and Data Mining, eds. U. Fayyad, G. Piatet-sky-Shapiro, P. Smyth, and R. Uthurusamy,447–471. Menlo Park, Calif.: AAAI Press.Horvitz, E., and Jensen, F. 1996. Proceedings of theTwelfth Conference of Uncertainty in Artificial Intelli-gence. San Mateo, Calif.: Morgan Kaufmann.Jain, A. K., and Dubes, R. C. 1988. Algorithms forClustering Data. Englewood Cliffs, N.J.: Prentice-Hall.Kloesgen, W. 1996. A Multipattern and Multistrate-gy Discovery Assistant. In Advances in KnowledgeDiscovery and Data Mining, eds. U. Fayyad, G. Piatet-sky-Shapiro, P. Smyth, and R. Uthurusamy,249–271. Menlo Park, Calif.: AAAI Press.Kloesgen, W., and Zytkow, J. 1996. Knowledge Dis-covery in Databases Terminology. In Advances inKnowledge Discovery and Data Mining, eds. U. Fayyad,G. Piatetsky-Shapiro, P. Smyth, and R. Uthurusamy,569–588. Menlo Park, Calif.: AAAI Press.Kolodner, J. 1993. Case-Based Reasoning. San Fran-cisco, Calif.: Morgan Kaufmann.Langley, P., and Simon, H. A. 1995. Applications ofMachine Learning and Rule Induction. Communica-tions of the ACM 38:55–64.Major, J., and Mangano, J. 1995. Selecting amongRules Induced from a Hurricane Database. Journalof Intelligent Information Systems 4(1): 39–52.Manago, M., and Auriol, M. 1996. Mining for OR.ORMS Today (Special Issue on Data Mining), Febru-ary, 28–32.Mannila, H.; Toivonen, H.; and Verkamo, A. I.1995. Discovering Frequent Episodes in Sequences.In Proceedings of the First International Confer-ence on Knowledge Discovery and Data Mining(KDD-95), 210–215. Menlo Park, Calif.: American

U. Fayyad, G. Piatetsky-Shapiro, P. Smyth, and R.Uthurusamy, 73–95. Menlo Park, Calif.: AAAI Press.Cheng, B., and Titterington, D. M. 1994. NeuralNetworks—A Review from a Statistical Perspective.Statistical Science 9(1): 2–30.Codd, E. F. 1993. Providing OLAP (On-Line Analyti-cal Processing) to User-Analysts: An IT Mandate. E.F. Codd and Associates.Dasarathy, B. V. 1991. Nearest Neighbor (NN)Norms: NN Pattern Classification Techniques.Washington, D.C.: IEEE Computer Society.Djoko, S.; Cook, D.; and Holder, L. 1995. Analyzingthe Benefits of Domain Knowledge in SubstructureDiscovery. In Proceedings of KDD-95: First Interna-tional Conference on Knowledge Discovery andData Mining, 75–80. Menlo Park, Calif.: AmericanAssociation for Artificial Intelligence.Dzeroski, S. 1996. Inductive Logic Programming forKnowledge Discovery in Databases. In Advances inKnowledge Discovery and Data Mining, eds. U.Fayyad, G. Piatetsky-Shapiro, P. Smyth, and R.Uthurusamy, 59–82. Menlo Park, Calif.: AAAI Press.Elder, J., and Pregibon, D. 1996. A Statistical Per-spective on KDD. In Advances in Knowledge Discov-ery and Data Mining, eds. U. Fayyad, G. Piatetsky-Shapiro, P. Smyth, and R. Uthurusamy, 83–116.Menlo Park, Calif.: AAAI Press.Etzioni, O. 1996. The World Wide Web: Quagmireor Gold Mine? Communications of the ACM (SpecialIssue on Data Mining). November 1996. Forthcom-ing.Fayyad, U. M.; Djorgovski, S. G.; and Weir, N. 1996.From Digitized Images to On-Line Catalogs: DataMining a Sky Survey. AI Magazine 17(2): 51–66.Fayyad, U. M.; Haussler, D.; and Stolorz, Z. 1996.KDD for Science Data Analysis: Issues and Exam-ples. In Proceedings of the Second InternationalConference on Knowledge Discovery and DataMining (KDD-96), 50–56. Menlo Park, Calif.: Amer-ican Association for Artificial Intelligence.Fayyad, U. M.; Piatetsky-Shapiro, G.; and Smyth, P.1996. From Data Mining to Knowledge Discovery:An Overview. In Advances in Knowledge Discoveryand Data Mining, eds. U. Fayyad, G. Piatetsky-Shapiro, P. Smyth, and R. Uthurusamy, 1–30. Men-lo Park, Calif.: AAAI Press.Fayyad, U. M.; Piatetsky-Shapiro, G.; Smyth, P.; andUthurusamy, R. 1996. Advances in Knowledge Dis-covery and Data Mining. Menlo Park, Calif.: AAAIPress.Friedman, J. H. 1989. Multivariate Adaptive Regres-sion Splines. Annals of Statistics 19:1–141.Geman, S.; Bienenstock, E.; and Doursat, R. 1992.Neural Networks and the Bias/Variance Dilemma.Neural Computation 4:1–58.Glymour, C.; Madigan, D.; Pregibon, D.; andSmyth, P. 1996. Statistics and Data Mining. Com-munications of the ACM (Special Issue on Data Min-ing). November 1996. Forthcoming.Glymour, C.; Scheines, R.; Spirtes, P.; Kelly, K. 1987.Discovering Causal Structure. New York: Academic.Guyon, O.; Matic, N.; and Vapnik, N. 1996. Discov-

Articles

52 AI MAGAZINE

Association for Artificial Intelligence.Matheus, C.; Piatetsky-Shapiro, G.; and McNeill, D.1996. Selecting and Reporting What Is Interesting:The KEfiR Application to Healthcare Data. In Ad-vances in Knowledge Discovery and Data Mining, eds.U. Fayyad, G. Piatetsky-Shapiro, P. Smyth, and R.Uthurusamy, 495–516. Menlo Park, Calif.: AAAIPress. Pearl, J. 1988. Probabilistic Reasoning in IntelligentSystems. San Francisco, Calif.: Morgan Kaufmann.Piatetsky-Shapiro, G. 1995. Knowledge Discoveryin Personal Data versus Privacy—A Mini-Sympo-sium. IEEE Expert 10(5).Piatetsky-Shapiro, G. 1991. Knowledge Discoveryin Real Databases: A Report on the IJCAI-89 Work-shop. AI Magazine 11(5): 68–70. Piatetsky-Shapiro, G., and Matheus, C. 1994. TheInterestingness of Deviations. In Proceedings ofKDD-94, eds. U. M. Fayyad and R. Uthurusamy.Technical Report WS-03. Menlo Park, Calif.: AAAIPress.Piatetsky-Shapiro, G.; Brachman, R.; Khabaza, T.;Kloesgen, W.; and Simoudis, E., 1996. An Overviewof Issues in Developing Industrial Data Mining andKnowledge Discovery Applications. In Proceedingsof the Second International Conference on Knowl-edge Discovery and Data Mining (KDD-96), eds. J.Han and E. Simoudis, 89–95. Menlo Park, Calif.:American Association for Artificial Intelligence.Quinlan, J. 1992. C4.5: Programs for Machine Learn-ing. San Francisco, Calif.: Morgan Kaufmann.Ripley, B. D. 1994. Neural Networks and RelatedMethods for Classification. Journal of the Royal Sta-tistical Society B. 56(3): 409–437. Senator, T.; Goldberg, H. G.; Wooton, J.; Cottini, M.A.; Umarkhan, A. F.; Klinger, C. D.; Llamas, W. M.;Marrone, M. P.; and Wong, R. W. H. 1995. The Fi-nancial Crimes Enforcement Network AI System(FAIS): Identifying Potential Money Launderingfrom Reports of Large Cash Transactions. AI Maga-zine 16(4): 21–39.Shrager, J., and Langley, P., eds. 1990. Computation-al Models of Scientific Discovery and Theory Forma-tion. San Francisco, Calif.: Morgan Kaufmann.Silberschatz, A., and Tuzhilin, A. 1995. On Subjec-tive Measures of Interestingness in Knowledge Dis-covery. In Proceedings of KDD-95: First Interna-tional Conference on Knowledge Discovery andData Mining, 275–281. Menlo Park, Calif.: Ameri-can Association for Artificial Intelligence.Silverman, B. 1986. Density Estimation for Statisticsand Data Analysis. New York: Chapman and Hall.Simoudis, E.; Livezey, B.; and Kerber, R. 1995. UsingRecon for Data Cleaning. In Proceedings of KDD-95:First International Conference on Knowledge Discov-ery and Data Mining, 275–281. Menlo Park, Calif.:American Association for Artificial Intelligence.Smyth, P.; Burl, M.; Fayyad, U.; and Perona, P.1996. Modeling Subjective Uncertainty in ImageAnnotation. In Advances in Knowledge Discovery andData Mining, 517–540. Menlo Park, Calif.: AAAIPress.

Spirtes, P.; Glymour, C.; and Scheines, R. 1993.Causation, Prediction, and Search. New York:Springer-Verlag.Stolorz, P.; Nakamura, H.; Mesrobian, E.; Muntz, R.;Shek, E.; Santos, J.; Yi, J.; Ng, K.; Chien, S.; Me-choso, C.; and Farrara, J. 1995. Fast Spatio-Tempo-ral Data Mining of Large Geophysical Datasets. InProceedings of KDD-95: First International Confer-ence on Knowledge Discovery and Data Mining,300–305. Menlo Park, Calif.: American Associationfor Artificial Intelligence.Titterington, D. M.; Smith, A. F. M.; and Makov, U.E. 1985. Statistical Analysis of Finite-Mixture Distribu-tions. Chichester, U.K.: Wiley.U.S. News. 1995. Basketball’s New High-Tech Guru:IBM Software Is Changing Coaches’ Game Plans.U.S. News and World Report, 11 December.Weigend, A., and Gershenfeld, N., eds. 1993. Pre-dicting the Future and Understanding the Past. Red-wood City, Calif.: Addison-Wesley.Weiss, S. I., and Kulikowski, C. 1991. Computer Sys-tems That Learn: Classification and Prediction Meth-ods from Statistics, Neural Networks, Machine Learn-ing, and Expert Systems. San Francisco, Calif.:Morgan Kaufmann.Whittaker, J. 1990. Graphical Models in Applied Mul-tivariate Statistics. New York: Wiley.Zembowicz, R., and Zytkow, J. 1996. From Contin-gency Tables to Various Forms of Knowledge inDatabases. In Advances in Knowledge Discovery andData Mining, eds. U. Fayyad, G. Piatetsky-Shapiro, P.Smyth, and R. Uthurusamy, 329–351. Menlo Park,Calif.: AAAI Press.

Usama Fayyad is a senior re-searcher at Microsoft Research.He received his Ph.D. in 1991from the University of Michiganat Ann Arbor. Prior to joining Mi-crosoft in 1996, he headed theMachine Learning Systems Groupat the Jet Propulsion Laboratory(JPL), California Institute of Tech-

nology, where he developed data-mining systemsfor automated science data analysis. He remainsaffiliated with JPL as a distinguished visiting scien-tist. Fayyad received the JPL 1993 Lew Allen Awardfor Excellence in Research and the 1994 NationalAeronautics and Space Administration ExceptionalAchievement Medal. His research interests includeknowledge discovery in large databases, data min-ing, machine-learning theory and applications, sta-tistical pattern recognition, and clustering. He wasprogram cochair of KDD-94 and KDD-95 (the FirstInternational Conference on Knowledge Discoveryand Data Mining). He is general chair of KDD-96,an editor in chief of the journal Data Mining andKnowledge Discovery, and coeditor of the 1996 AAAIPress book Advances in Knowledge Discovery and Da-ta Mining.

Articles

FALL 1996 53

cal Engineering Departments at Caltech (1994) andregularly conducts tutorials on probabilistic learn-ing algorithms at national conferences (includingUAI-93, AAAI-94, CAIA-95, IJCAI-95). He is generalchair of the Sixth International Workshop on AIand Statistics, to be held in 1997. Smyth’s researchinterests include statistical pattern recognition, ma-chine learning, decision theory, probabilistic rea-soning, information theory, and the application ofprobability and statistics in AI. He has published 16journal papers, 10 book chapters, and 60 confer-ence papers on these topics.

Gregory Piatetsky-Shapiro is aprincipal member of the technicalstaff at GTE Laboratories and theprincipal investigator of theKnowledge Discovery in Databas-es (KDD) Project, which focuseson developing and deploying ad-vanced KDD systems for businessapplications. Previously, he

worked on applying intelligent front ends to het-erogeneous databases. Piatetsky-Shapiro receivedseveral GTE awards, including GTE’s highest tech-nical achievement award for the KEfiR system forhealth-care data analysis. His research interests in-clude intelligent database systems, dependencynetworks, and Internet resource discovery. Prior toGTE, he worked at Strategic Information develop-ing financial database systems. Piatetsky-Shapiro re-ceived his M.S. in 1979 and his Ph.D. in 1984, bothfrom New York University (NYU). His Ph.D. disser-tation on self-organizing database systems receivedNYU awards as the best dissertation in computerscience and in all natural sciences. Piatetsky-Shapiro organized and chaired the first three (1989,1991, and 1993) KDD workshops and helped in de-veloping them into successful conferences (KDD-95and KDD-96). He has also been on the programcommittees of numerous other conferences andworkshops on AI and databases. He edited andcoedited several collections on KDD, including twobooks—Knowledge Discovery in Databases (AAAIPress, 1991) and Advances in Knowledge Discovery inDatabases (AAAI Press, 1996)—and has many otherpublications in the areas of AI and databases. He isa coeditor in chief of the new Data Mining andKnowledge Discovery journal. Piatetsky-Shapirofounded and moderates the KDD Nuggets electronicnewsletter (kdd@gte.com) and is the web master forKnowledge Discovery Mine (<http://info.gte.com/~kdd /index.html>).

Padhraic Smyth received a first-class-honors Bachelor of Engi-neering from the National Uni-versity of Ireland in 1984 and anMSEE and a Ph.D. from the Elec-trical Engineering Department atthe California Institute of Tech-nology (Caltech) in 1985 and1988, respectively. From 1988 to

1996, he was a technical group leader at the JetPropulsion Laboratory (JPL). Since April 1996, hehas been a faculty member in the Information andComputer Science Department at the University ofCalifornia at Irvine. He is also currently a principalinvestigator at JPL (part-time) and is a consultant toprivate industry. Smyth received the Lew AllenAward for Excellence in Research at JPL in 1993and has been awarded 14 National Aeronautics andSpace Administration certificates for technical in-novation since 1991. He was coeditor of the bookAdvances in Knowledge Discovery and Data Mining(AAAI Press, 1996). Smyth was a visiting lecturer inthe Computational and Neural Systems and Electri-

Articles

54 AI MAGAZINE

AAAI 97Providence, Rhode Island

July 27–31, 1997

Title pages due January 6, 1997

Papers due January 8, 1997

Camera copy due April 2, 1997

ncai@aaaai.org

http://www.aaai.org/

Conferences/National/1997/aaai97.html