rule-based method for entity resolution ieee transactions on knowledge and data engineering january...

Rule-Based Method for Entity Resolution

IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING

JANUARY 2015

INTRODUCTIONIN many applications, a real-world entity may appear inmultiple data sources so that the entity may have quitedifferent descriptions. For example, there are severalways to represent a person’s name or a mailing address.Thus, it is necessary to identify the records referring tothe same real-world entity, which is called Entity Resolution(ER). ER is one of the most important problemsin data cleaning and arises in many applications suchas information integration and information retrieval.Because of its importance, it has attracted much attentionin the literature

• Traditional ER approachesSimilarity comparison among records.

Can’t identify records correctly in some cases.

observation:The existence and nonexistence of some attribute-value pairs are both useful to identify records

Contribution

syntax

semantics

Properties of ER-Rule Set

Algorithm

• Rule Discovery(DiscR) -To get rules from a training data set• Rule-based entity resolution (R-ER) -To determine the record in the new data set refers to which entity

Rule Discovery

• Several definition before the algorithm

Rule Discovery

• Rule requirements

Gen-PR

Gen-SingleNRFirst step:

Second step:

Rule-based entity resolution

• we define the weight of each ER-rule r as:

Rule update

• Invalid rules• Useless rules

Evaluation• the effectiveness of our rule learning algorithm (DiscR) and

our rule-based ER approach• the impact of training data size on ER accuracy and the

number of generated rules• The impact of rule length threshold on ER accuracy• The scalability of DiscR and R-ER with the size of data

• Algorithm compared with: GHOST and CFR

Summary• DiscR and R-ER can achieve a high accuracy using a small

training data;• updating rules indeed help identify records; • The number of generated rules scales well with the training

data size on both data sets; • rules with length larger than 2 are seldom needed to identify

records; • both DiscR and R-ER scales well with the size of data.

Thank you!