1

Quality Control for Field Operations

2

Overview

Goal• To ensure the quality of survey field

work

Purpose• To detect and deter interviewer errors

and falsification

3

Overview (cont.)

What do we check in QC for field work?• Interviewer procedures• Forms• Interviewer productivity• Interviews• Address listing

4

Overview (cont.)

What are the tools for performing QC on field work?• Interviewer observations• Formal / Informal reviews• Monitoring reports• Reinterview• Address listing QC

5

Interviewer Observations

What are the goals of interviewer observations?• To assess effectiveness of interviewer

training

• To ensure interviewers understand and follow prescribed interviewing procedures

• To provide interviewers with feedback to encourage continuous improvement

6

Interviewer Observations (cont.)

What are interviewer observations?• Supervisor “shadows” interviewer during

initial assignments• Observe interviewer’s work to ensure

appropriate procedures are followed• Evaluate interviewer’s work and provide

positive and negative feedback to interviewer

• See Handout on Field Control Form Information

7

Interviewer Observations (cont.)

What are the results of observations?• Supervisor decides if interviewer’s

performance is acceptable

• If so, allow interviewer to continue work independently

• If not, re-train interviewer on procedures or release interviewer from the operation

8

Formal / Informal Reviews

What are the goals of reviews?

• To ensure the quality of the data collected on forms and questionnaires

• To ensure interviewers and clerks understand and follow procedures for completing forms and questionnaires

9

Formal / Informal Reviews (cont.)

How are informal reviews performed?• Supervisor meets with employees to

review forms and/or questionnaires

• Review for legibility, completeness, and accuracy

• Provide positive and negative feedback to employee for continuous improvement

• Repair errors where necessary

10

Formal / Informal Reviews (cont.)

How are formal reviews performed?• Usually performed in a field office by

office staff

• Review questionnaires or listings for legibility, completeness, and accuracy

• Repair errors or return materials to the field when necessary

11

Formal / Informal Reviews (cont.)

What work products are subject to reviews?

• Any form or questionnaire that requires a quality check (e.g., payroll forms, questionnaires, address listing forms, etc.)

12

Formal / Informal Reviews (cont.)

What are the results of informal reviews?• Supervisor determines if the form or

questionnaire meets quality standards

• If so, the form/questionnaire is accepted and submitted

• If not, repair the form/questionnaire prior to submission

13

Monitoring Reports

What are the goals of monitoring reports?

• To track status of operations

• To inform management of progress

• To identify problems early for timely rectification

14

Monitoring Reports (cont.)

How do we know what reports to monitor for QC purposes?• Define quality goals for the operation (e.g.,

response rates, productivity rates, cost, etc.)

• Identify variables that provide information on quality goals

• Design report(s) that present the important variables for quick review and assessment

15

Monitoring Reports (cont.)

How do we monitor the reports for QC purposes?

• Determine necessary frequency for monitoring (usually daily)

• Look for outlier values (e.g., low response rates, high costs, etc.)

• Determine reason(s) for outlier values

• Take appropriate action

16

Reinterview

What is reinterview?

• Quality check on interviewing activities

• Primary tool for ensuring quality of interviewers’ field work

• Independent, objective review of interviewing work

17

Reinterview (cont.)

What types of reinterview can be used?

• Random reinterview

• Administrative reinterview

• Supplemental reinterview

18

Reinterview (cont.)

What decisions are made in reinterview?• Falsification (release interviewer)

• Interviewer error (retraining/feedback)

• Respondent error (no action)

• Rework may be necessary for falsification or other serious errors

19

Random Reinterview

What is the purpose of the random reinterview component?

• Provides broad protection over entire workload and all interviewers

• Serves as an effective deterrent for interviewer falsification

20

Random Reinterview (cont.)

What is the sampling design for the random reinterview component?• Select random sample of cases within

every enumerator’s workload, or

• Two-stage design: take a sample of interviewers, then a sample of cases within those interviewers’ workloads

21

Administrative Reinterview

What is the purpose of the administrative reinterview component?

• Provides “targeted” approach to identifying suspect work

• Provides valuable performance data to field supervisors

22

Administrative Reinterview (cont.)

How does the administrative model work?• Accumulates data on work

characteristics for all interviewers• Vacant housing units• Partial interviews• Single-person households• Deleted housing units• Average population per housing unit

23

Administrative Reinterview (cont.)

How does the administrative model work? (cont.)

• Compare accumulated data for each interviewer to the data for all other interviewers in their assignment area

• Identify and flag interviewers who are significantly different than their area average

24

Administrative Reinterview (cont.)

How does the administrative model work? (cont.)

• Provide field supervisors with a list of “outlier” interviewers and what variables caused the flag

• Field supervisors determine if the interviewers’ “outlier” status is explicable

• More of suspect interviewer’s cases put into reinterview

25

Supplemental Reinterview

What is the purpose of the supplemental reinterview?

• Provides field supervisors with the capability of checking work for selected interviewers

• Tool for use during investigation of suspected falsification

26

Supplemental Reinterview (cont.)

What is the sampling design for the supplemental reinterview?

• Select a random sample of cases from the interviewer’s workload

• Field supervisor can specify how many cases to check

27

Address Listing QC

What is the goal of address listing QC? To ensure the quality of address

listing work

• To protect against undercoverage and overcoverage

28

Address Listing QC (cont.)

What are the tools for performing address listing QC?

Dependent QC

• Advance listing

• Suppression

29

Address Listing QC (cont.)

What decisions are made in address listing QC?

• If errors exceed acceptable limit, recanvass assignment area

• Repair minor mistakes

• Listers may be released or retrained depending on severity of errors

30

Dependent QC

What is dependent QC?

• Random, independent check of addresses listed by survey listers

• Field supervisors verify existence of housing units or validity of housing unit deletes

31

Dependent QC (cont.)

What is the sampling design for dependent QC?

• Select a random sample of housing units to verify from the listers’ workload

• Sample should include all types of listings (e.g., verifications, adds, deletes)

32

Advance Listing

What is advance listing?

• Field supervisor does initial listing of assignment areas prior to making assignments to listers

• Gives supervisor knowledge of the area so they are able to detect faulty or inadequate work

33

Suppression

What is suppression?

• Selected addresses are purposely left off listings

• Listers’ ability to find and add the suppressed addresses is assessed

• Provides confidence in the listers’ ability to perform adequate canvassing