Quality Control Issues During Data Analysis

Demographic Analysis

Tom McDevitt, Population Studies Branch, International Programs Center,

U.S. Census Bureau

Anatomy of a Survey/Census - 5 Phases

Contract Negotiation

Design and Development

DataCollection

Post-CollectionProcessing

Analysis andDissemination

Each phase has its own:• Objective• Key tasks• Deliverables• Documentation

The Data Analysis and Data Collection Link

• Data analysis generates information constrained by census/survey design.

• Data evaluation identifies limitations.

• Census/survey design builds on the previous two operations.

Quality Control Issues During Data Analysis

• Strategies for Identifying Errors and Adjusting International Demographic Data

• Strategies of the U.S. Census 2000 Plan

Pay now or pay later.

There is no such thing as a free lunch.

Pay now or pay later.

Strategies for Identifying

Errors and Adjusting

International

Demographic Data

Demographic data must be evaluated and a decision taken

Demographic data must be evaluated and a decision taken:

Accept the data without modification.

Demographic data must be evaluated and a decision taken:

Accept the data without modification.

Adjust the data.

Syria: 19941. Population by Age and Sex

0

200,000

400,000

600,000

800,000

1,000,000

1,200,0000

200,000

400,000

600,000

800,000

1,000,000

1,200,000

0-4

10-14

20-24

30-34

40-44

50-54

60-64

70-74

80+Male Female

Checking data quality using a population pyramid

Syria: 1994, 1981, 1970 Censuses

10,000

100,000

1,000,000

10,000,000

18601880190019201940196019802000Year of Birth (5-year cohorts)

Popu

latio

n

1970 1981 1994

Male Population, 5-Year Cohorts

Checking inter-censal consistency of data with cohort analysis

Quality Control in Analysis

• Product control– In order to control quality in census/survey analysis, we need

definitions of what is acceptable for each product, decision rules to determine which products are accepted or rejected, and appropriate actions to take based on the results of the decision.

• Process control– Control the methods used to monitor the operation.

– Control the steps that determine product acceptance and, in the extreme, when an employee needs to be retrained or released.

Quality Control Systems, Demographic Analysis at IPC

• Methodology standardization – PAM & PAS• Standardized documentation procedures

– Within work files

– Paper documentation

• Verification• Supervision• Review of analyst’s work by others

Quality Control Points in Demographic Analysis at IPC

• Analyst -- data re-entry and analysis

Quality Control Points in Demographic Analysis at IPC

• Analyst -- data re-entry and analysis• Verifier 100% verification

Quality Control Points in Demographic Analysis at IPC

• Analyst -- data re-entry and analysis• Verifier 100% verification• E&P country coordinator• Branch chief• Senior demographer

Quality Control Points in Demographic Analysis at IPC

• Analyst -- data re-entry and analysis• Verifier 100% verification• E&P country coordinator• Branch chief• Senior demographer

Quality Control Points in Demographic Analysis at IPC

• Analyst -- data re-entry and analysis• Verifier 100% verification• E&P country coordinator• Branch chief• Senior demographer• Author • Statistical review• Publications and printing• Author Product

Strategies of the

U.S. Census 2000 Plan

Strategies of the

U.S. Census 2000 Plan

1. Build partnerships.

2. Keep it simple.

3. Use technology intelligently.

4. Use statistical methods.

Alternative strategies exist for addressing the issue of the quality of demographic data, whether these be from the United States or from another country.

It's a matter of choice:

Do it right the first time. Collect data using procedures that minimize error and maximize data quality.

Attempt to correct the data, to compensate for error, at the analysis stage.

(Work with data collection staff to improve data collection next time).

Quality in Demographic Data

Alternative strategies exist for addressing the issue of the quality of demographic data, whether these be from the United States or from another country.

The choices are essentially 3 in number:

Do it right the first time. Collect data using procedures that minimize error and maximize data quality.

Attempt to correct the data, to compensate for error, at the analysis stage.

Ignore data quality

Quality in Demographic Data

Lessons of Experience• In demographic analysis, use multiple

techniques and compare results.• Use multiple datasets where possible,

and recognize that reference “standards” must also be evaluated.

• Graph data.• Verification – “Trust, but verify.”• Monitor and review an analyst’s work.• Learn from experience, and provide

feedback to the data collection planning effort.

Two Pathways and the Goals of Data Quality:

Relevance

Accuracy

Timeliness

Accessibility

Interpretability

Coherence

There is no such thing as a free lunch.

Pay now or pay later. . .

. . . In most instances, in order to obtain good population data, it is cheaper, easier, better to pay “up front.”