D E R E K H A N S E N , J A K E G E H R I N G , PAT R I C K S C H O N E , A N D M AT T H E W R E I D

FAMILY HISTORY TECHNOLOGY WORKSHOPFEBRUARY 3, 2012

IMPROVING INDEXING EFFICIENCY & QUALITY:COMPARING A-B-ARBITRATE AND PEER REVIEW

FAMILYSEARCH

FAMILYSEARCH INDEXING

A-B-ARBITRATE PROCESS (A-B-ARB)

A

B

ARB

THE PROBLEM

OUR APPROACH•Historical Data Analysis•Field Experiment comparing quality control models

HISTORICAL DATA ANALYSIS• Quality (estimated based on A-B agreement)• Measures difficulty more than actual quality• Underestimates quality, since an experienced Arbitrator

reviews all A-B disagreements• Good at capturing differences across people, fields, and

projects• Time (calculated using keystroke-logging data)• Idle time is tracked separately, making actual time

measurements more accurate• Outliers removed

A-B AGREEMENT BY FIELD

A-B AGREEMENT BY LANGUAGE

English Language• Given Name: 79.8• Surname: 66.4

French Language• Given Name: 62.7%• Surname: 48.8%

1871 Canadian Census

A-B AGREEMENT BY EXPERIENCE

Birth Place: All U.S. CensusesB

(nov

ice ↔

exp

ert)

A (novice ↔ expert)

A-B AGREEMENT BY EXPERIENCE

Given Name: All U.S. CensusesB

(nov

ice ↔

exp

ert)

A (novice ↔ expert)

A-B AGREEMENT BY EXPERIENCE

Surname: All U.S. CensusesB

(nov

ice ↔

exp

ert)

A (novice ↔ expert)

A-B AGREEMENT BY EXPERIENCE

Gender: All U.S. CensusesB

(nov

ice ↔

exp

ert)

A (novice ↔ expert)

A-B AGREEMENT BY EXPERIENCEU.S. - English Canada - English

Canada - FrenchMexico - Spanish

TIME & KEYSTROKE BY EXPERIENCE

TIME & KEYSTROKE OF ARB

A NEW APPROACH? (A-R-ARB)

• Peer review model• Efficiency ++•Quality ?

PEER REVIEW PROCESS (A-R-ARB)

A R ARB

Already Filled In Optional?

FIELD EXPERIMENT

• Develop Truth Set of 2,000 1930 Census images• Use historical A-B-ARB data• Create new A-R-ARB dataset by having

new indexers review and arbitrate• Compare quality & efficiency• Qualitatively identify types of errors

DISCUSSIONIMPLICATIONS• Transition users from novice to expert• Recruit foreign language indexers• Intelligent matching based on expertise

(in A-B-ARB &/or A-R-ARB)

FUTURE POSSIBILITIES• Peer review by algorithms?• Initial indexing by algorithms?