Multi-Mode Data Collection: Why, When, How

International Conference on Establishment Surveys

MontrealJune 18-21, 2007

Richard RosenUS. Bureau of Labor Statistics

rosen.richard@bls.gov202-691-6524

Any opinions expressed in this paper are those of the authors and do not constitute policy of the Bureau of Labor Statistics.

Automated Collection in the Current Employment Automated Collection in the Current Employment Statistics SurveyStatistics Survey

Voice Recognition (VR)

CATI

Touchtone Data Entry (TDE)Computer Assisted Telephone

Interviewing (CATI)

FAX

Data Collection Centers (DCC)

Web Data Collection

Electronic Data Interchange (EDI)

Kansas CityChicago

Dallas

Atlanta

CES Data

RespondentDatabase

BLS

Goals for Paper To provide insights/experiences

based on CES To review the collective experience

of all BLS Establishment-based surveys

To draw conclusions about the relative benefits of multi-mode collection

Current Employment Statistics Current Employment Statistics ProgramProgram

Monthly survey of employment, payroll, and hours conducted by US Bureau of Labor Statistics

A Federal-State cooperative system

A sample of 300,000 business establishments

Data are published after only 12 collection days

Limited number of data elements (but greatly increased in 2006)

Multi-mode since 1984

Automated Collection Methods in the CESAutomated Collection Methods in the CES

In 1984, BLS began to examine alternative collection methods.

New automated collection modes: YearComputer Assisted Telephone Interview (CATI) 1984

Touchtone Data Entry (TDE) 1986

Voice Recognition (VR) 1988

Electronic Data Interchange (EDI) 1994

FAX 1995

Electronic Mail/Internet/WWW 1996

Multi-mode Myths

It will lower my collection cost It will improve my response rate Implementation will be easy It will solve all of my problems

Cost

Depends on what mode is being replaced

Development costs Initial start-up costs Economies of scale Cost structure change over time

Estimated CES Collection Costs by Mode

Mode: Unit cost per month:

CATI $6.43

Mail $2.10

FAX $1.60

TDE $1.50

Web $0.80

EDI $0.50

Response Rates

Depends on the mode being replaced Mail vs any mode = improved response Electronic Data Interchange = generally

lower initial response; time lag CATI can achieve high response rates TDE, FAX, Web, E-mail: Self-response

Lower response than CATI Higher response than Mail

Easy

Requires R&D Acquisition of new HW/SW IT Support Prototype Testing

Solve All Problems

Create new set of problems New protocols Integration with other collection

modes Integration with other survey

operations Impact on current staffing/workflow

TDE Respondent Contact ProgramTDE Respondent Contact Program

Contact Type: Contact Method: Advance Notice FAX or Postcard Nonresponse Prompt FAX or Call “Last Chance” prompt FAX Secondary NRP Call Long-term NRP Call Refusal Conversion Letter/Call

CES has implemented procedures designed to maximize response

Example of Protocol Effect Effect of Advance Notice and

Nonresponse prompting on TDE Response Elimination of Advance Notice:

Reduced response by 10 % points Elimination of Nonresponse message:

Reduced response by 10 % points

Rosen and Hertwig, “The Impact of Prompting on Response Rates: Experience with Touchtone Reporting in the CES Program,” American Statistical Association, August 2002.

Comparison of CES Collection MethodologiesComparison of CES Collection Methodologies

CollectionMethod

DataReporting Procedures

Monthly Advance Notice

Non-ResponsePrompting

TouchtoneData Entry

(TDE)

Respondent calls toll free 800 number, connects to computer, enters data using touchtone phone

Postcard or FAX

-Phone call (if large)-FAX (if small with FAX)-Postcard (if small, no FAX)

CATIRespondent called by interviewer, data entered into computer

PostcardCATI call serves as prompt

FAXRespondent FAXes completed form

Blank Form-Re-FAX survey form-Phone call

WebRespondent links to

secure web site, enters data and submits

E-mailE-mail

Mail Respondent mails completed form

Blank Form

-Re-mail survey form-Postcard-Letter-Phone call

Electronic Data

Interchange (EDI)

Respondent creates a file which is transmitted to BLS electronically

----Personal phone call

What Factors to Consider? Characteristics of your survey

Periodicity Survey length/complexity Sample composition

Characteristics of your respondents Knowledge/education Environment (office vs mobile) Commitment/willingness to report

Distribution of CES Sample byDistribution of CES Sample byCollection ModeCollection Mode

1 6

2 5

3 4

4 3

5 2

6 1

Distribution of CES Sample by Collection Mode

March 2000 May 2003Web 1%

EDI 6%

Tape/disk 7%

FAX 7% Mail 16% CATI 5%

TDE 59%

Web 1%

TDE 33%

CATI 18%

Mail 4%

Tape/disk 6%

EDI 23%

FAX 15%

Comparison of CES Collection Rate, Employment Received, and Revisions

(Nov. 1988-Jan. 2001)

0

10

20

30

40

50

60

70

80

90

100

Nov

-88

May

-89

Sep

-89

Jan-

90

Jul-9

0

Nov

-90

May

-91

Sep

-91

Jan-

92

Jul-9

2

Nov

-92

May

-93

Sep

-93

Jan-

94

Jul-9

4

Nov

-94

May

-95

Sep

-95

Jan-

96

Jul-9

6

Jan-

97

Jul-9

7

Jan-

98

Jul-9

8

Jan-

99

Jul-9

9

Jan-

00

Jul-0

0

Jan-

01

Date

Pe

rce

nt

0

20

40

60

80

100

120

140

160

Ab

so

lute

Va

lue

of

Re

vis

ion

s

(10

00

)

Percent Employment Received at First Closing Percentage of Registry Received at First Closing

Absolute Value of Total Private Revisions from 1st-3rd Closing

Profile of CES Population and Collection Profile of CES Population and Collection MethodsMethods

Range ofworksites

Number of firms(EINs)

Percent offirms (EINs)

Percent ofworksites

Percent ofEmployment

1 - 4 208,831 93.6% 29.4% 39.2%

5 - 9 6,365 2.9% 5.2% 9.0%

10 - 19 3,826 1.7% 6.5% 7.4%

20 - 49 2,326 1.0% 8.8% 9.5%

50 - 99 820 0.4% 7.1% 7.3%

100 - 249 559 0.3% 11.0% 8.2%

250 - 499 192 0.1% 8.2% 5.2%

500+ 139 0.1% 23.6% 14.1%

Total 223,058 100.0 100.0% 100.0%

TDE/Web/E-mail Fax/XLS/Fillable Form EDI

CES Collection Rates for Preliminary Estimates by Mode J un 03 - Mar 07

0102030405060708090

100Ju

n-03

Oct

-03

Feb

-04

Jun-

04

Oct

-04

Feb

-05

Jun-

05

Oct

-05

Feb

-06

Jun-

06

Oct

-06

Feb

-07

Co

llect

ion

Ra

te

Total Mail TDE CATI Fax Web EDI

CES Collection Rates by Mode, Final Estimates. Jun 2003 - Mar 2007

0.0

10.0

20.0

30.0

40.0

50.0

60.0

70.0

80.0

90.0

100.0

Co

llec

tio

n R

ate

Total Mail TDE CATI FAX Web EDI

Web Collection: Some Advice KISS (keep it simple stupid)

Respondents are not sophisticated computer users Beware of overbearing security requirements

Digital Certificates, complicated passwords, passwords that must be updated, pose significant barrier

Keep edits simple Don’t try to replicate all of your edit checks on-line Edit failures when data are correct will frustrate

respondents In April 2007, BLS launched a more streamlined

Website for data reporting (Web-lite).

TDE vs Web Response Rates for Preliminary Estimates, Jan. 2004 - Mar. 2007

40%

50%

60%

70%

80%

90%

100%

Jan0

4

Feb04

Mar

04

Apr04

May

04

Jun04

Jul0

4

Aug04

Sep04

Oct04

Nov04

Dec04

Jan0

5

Feb05

Mar

05

Apr05

May

05

Jun05

Jul0

5

Aug05

Sep05

Oct05

Nov05

Dec05

Jan0

6

Feb06

Mar

06

Apr06

May

06

Jun06

Jul0

6

Aug06

Sep06

Oct06

Nov06

Dec06

Jan0

7

Feb07

Mar

07

TDE Web

Begin Web-lite April 2006

E-mail vs Web Easier for respondent

Eliminates login process Can’t forget your account number of password

Can be done securely with HTTPS Can embed HTML form or PDF directly into E-mail Respondent fills out form and hits a “submit”

button Data sent via Browser to agency server using

HTTPS Several products on market

E-mail Drawbacks Limited editing capability With PDF, size of file may be an issue With HTML,

Number of data items/questions is limited (single page)

Not all respondents have HTML E-mail Products available don’t offer “total

solution” so must develop back-end support

TDE vs Web vs E-mail Collection Rate Comparison

2007 All TDE Web E-mail

February

73% 73% 66% 74%

March 68% 73% 64% 67%

April 69% 69% *60% 70%

*CES experienced web server problems during April 2007

EDI: Some Comments

Takes time to work with firms Most Gov’t surveys are voluntary Must “get in line” for IT resources

Have a standard file format but be prepared to take what they have Data item response can be an issue

New directions: XML

Review of BLS Establishment Surveys

Eight surveys Four monthly Two Quarterly Two Annual

Over the past 10 years, all have adopted multi-mode collection

Survey Questions Distribution of modes used Reason for using multiple modes Do you target certain populations for

specific modes Benefits and Drawbacks Process used to determine new modes New modes being considered Key factors when considering new

modes Advice to other survey organizations

Distribution of Modes, BLS Establishment Surveys, 2007

 CES IPP JOLTS PPI MWR NCS SOII OES  

Mode: M M M M M Q A A Average

Mail 8% 41% 13% 54% 84% 19% 46% 85% 44%

CATI 23%   40%           8%

Fax 18%   18% 45%   13% 2% 1% 12%

Touchtone Data Entry 20%   28%           6%

Electronic data 30%   1% 1% 15% 26% 4% 2% 10%

Web 1% 56%     1%   46%   13%

Telephone           28% 3% 11% 5%

Personal Visit           5%     1%

Combination           10%     1%

Other   3%           3% 1%

M=Monthly Q=Quarterly A=Annual

Reasons Using Multi-mode

Reason for using multi-mode: CES JOLTS PPI IPP NCS MWR SOII OES Score

Respondent preference 3 1 1 1 2,3 3   3 39

Reduce cost 2       6 1 1   18

Improve response rates 1       1     1 18

Better Data Quality       5 5   2   9

Quicker Turnaround     3 4         7

Technology and resources available  

2             5

Problems with Mail delivery     2           5

Support GPEA       2         5

Reduce Respondent Burden           2     5

State Partner Feedback               2 5

Ease of Use for Respondent       3         4

Improve Productivity         4       3

Do you Target Modes? CES: Take into account the characteristics of the firm in terms of size, number of

units/reports, past reporting history in terms of timeliness and ability to self-report.

MWR: Large multi-state employers encourage to use EDI Center; small multi-unit respondents 2-30 worksites offered Web. Also target software developers and outsourcing firms to include electronic reporting in their systems or services.

NCS: Mode of collection is determined by size of establishment, location of establishment, reporting capability of the firm, and level of cooperation.

SOII: All modes are available to all respondents. Certain respondents get booklets

designed to encourage internet and other electronic methods or reporting.

IPP: During the visit respondents are told that web is the preferred mode of collection but that mail/fax is also available if that is their preference. IPP wants the respondent to choose whatever method they feel most comfortable with. Since September 2005, just over 70% of IPP respondents have selected web.

PPI: prefers electronic data collection, which currently is limited to Fax. For

important respondents, we allow them to email in spreadsheet with pricing data. JOLTS: With the exception of our largest respondents, we normally encourage our

respondents to provide data via TDE after a 6 month period of data collection using CATI.

OES: COCs or establishment with high weight; previous response via a particular mode; availability of email or web site addresses

Drawbacks CES: Time spent in researching and developing new mode. Need to develop new protocols

and procedures for each mode. Often end up with separate databases for each mode that need to be managed. Almost like running multiple/separate survey operations.

MWR: Requires sizable amount of staff resources to maintain systems and deal with coordination and timing issues. Keeping States and State systems and employers in sync can be a challenge.

NCS: It is virtually impossible to isolate and evaluate the effectiveness of any single mode. Have to maintain instructions and enforce protocols in each area. Additional tracking and maintenance is required to keep up with each firm’s collection mode. Need to customize update materials and data requests to match the mode used by each firm. Electronic collection carries risks of confidentiality breaches.

SOII: For some reason electronic collection has a slightly lower response rate than respondents receiving the standard booklets.

IPP: Additional costs/resources to maintain multiple modes.

PPI: The major drawback for PPI is the BLS limitations on email, not having web repricing and systems limitations on broadcast fax pricing.

JOLTS: None.

OES: Increased occupational coding burden on State analysts; Internal and external security/confidentiality issues; Costs.

Process to Determine New Mode

CES: Initial research on mode, possible use/benefit. Proof on concept project. If appears to be successful, limited production. Then full implementation. Constant evaluation and monitoring.

MWR: Evaluated pros and cons of various collection modes in terms of expected costs, accuracy, security, periodicity of collection. For example, if survey is only done every 3 years do you want to invest in new costly collection mode.

NCS: The last mode to be added, electronic collection through email to a secure server, arose due to demand from respondents and field economists. It was then developed and tested.

SOII: Does the new mode reduce cost and/or help capture additional narrative.

IPP: More respondents started requesting additional electronic alternatives, such as Internet-based collection. Initially piloted the web survey with a small sample of reporters. Results of the pilot were very favorable.

PPI: Broadcast fax was a pilot project for several years with limited respondents. When the anthrax problem disrupted mail pricing, then we started repricing forms by fax and gradually increased our broadcast fax capabilities

OES: Feedback; Research and IT consultation; Security review; Pilot testing and process refinement

New Modes Under Consideration

CES: E-mail collection either with embedded HTML form or fillable XLS/PDF

MWR: Fillable forms NCS: secure, encrypted data files SOII: None IPP: Fillable PDF survey forms via email PPI: Web repricing JOLTS: Secure E-mail OES: Fillable forms, Web

Factors to Implement CES: Ease of use for respondent; respondent

acceptance. Data security. MWR: Cost, employer acceptance, State

acceptance, employer familiarity with the survey. NCS: If the test is successful, it is relatively easy

to use, and respondents find it acceptable. IPP: Respondent demand, costs and other

measurable compared to IPPs current collection alternatives.

JOLTS: Cost of using the central BLS facility for secure email. Ease of use for respondent.

OES: Feasibility to collect meaningful data; Cost; Security

Study Mode Effects

CES: Mostly response rate effects and data item response. Some review of data quality.

MWR: Web collection has higher response rates, but this may be somewhat biased as we restricted initial solicitation to good reporters.

NCS: No

SOII: response rates, processing times

IPP: With web repricing the IPP gets substantially faster data turnaround than mail/fax; Clerical and quality edits on the front-end of the web survey yield better quality and more usable data than mail/fax; response rates have been consistently greater for web than mail/fax respondents.

PPI: No

JOLTS: No

OES: Response Analysis Survey will include review of mode effects

Advice CES: Surveys need to modernize their collection. Carefully consider alternative collection

modes, do the needed development work and evaluation. Don’t expect the new mode to solve all of your problems. If you can solve just one problem or meet one objective that may be enough. Constantly evaluate your “mix” as things change, and you have to adapt and add new modes and de-emphasize others.

MWR: Consider costs, employer familiarity with the survey, timing and coordination issues.

NCS: Evaluate how secure any new mode of collection will be and whether any policy precludes a specific mode from use. Assess what information is available in more than one mode and whether respondent burden or collection costs can be reduced, and whether efficiency or quality is improved by adding a new mode. Need to have technology support and staff training on any new collection mode employed.

SOII: Do it.

IPP: Know your respondents; get support from upper management; Iterative development and pilot testing; Usability testing a must; keep stakeholders informed; provide resources for ongoing project management.

PPI: Be prepared to budget in advance as developing electronic collection methods is costly.

OES: Talk to others to find out their lessons learned; establish and maintain a good working relationship with IT; lots of research; have a tech close by; Pilot testing; Training

Summary Most BLS surveys are using multi-mode Respondent preference/acceptance is top

priority Cost should NOT be primary goal

(secondary or side benefit) Improved response is a possible benefit Improved timeliness is possible/likely Additional modes add complexity to

survey operations Continuous evaluation of effectiveness;

can’t stand still