1

1

Web Web and Paper and Paper Questionnaires seen Questionnaires seen from the Business Respondent’s from the Business Respondent’s PerspectivePerspective

Gustav Haraldsen, Statistics Norway

Jacqui Jones, UK Office of National Statistics

Third International Conference on Establishment surveys. Montréal 2007

2

Outline

• A quality driven approach to response burden

• The effect of survey design, external conditions and personal characteristics on perceived response burden

• The impact of perceived response burden on data collection and data quality

• (The perceived response burden of web versus paper questionnaires)

3

Stakeholder (s)

Survey organisation

Survey Design

PRB

Respondent(s)• Availability• Motivation• Competence

Responsequality

Response environment Survey Requirements

QuestionnaireLengthStructureContentLayout

ProcedureContact mode(s)Persuasion strategiesManagementConfidentiality

Sample SpecificationFrameSampling procedureSize

Release

Origins and Effects of Perceived Response Burden in Establishment Surveys

Context:Encoding/record formationRespondent selection/identificationPriorities

Revision

4

The quality driven approach

• Response burden is mainly a problem as long as it affects the response quality

• It is the respondents’ response burdens that matters

• Perceived response burden is a more relevant variable than completion time

• The response burden may be affected by external conditions or personal characteristics of the respondents as well as by the survey design.

• Perceived response burden questions should identify why questionnaires are burdensome

5

Core PRB Questions Do you think it was quick or time consuming to collect the necessary information to complete the questionnaire?

Very quick

Quite quick

Neither quick nor time consuming These were asked to pick the main reason(s) from a list or specify other reason(s)

Quite time consuming

Very time consuming

Did you find it easy or burdensome to complete the questionnaire?

Very easy

Quite easy

Neither easy nor burdensome These were asked to pick the main reason(s) from a list or specify other reason(s)

Quite burdensome

Very burdensome

+1

+0.5

0

-0.5-1

+1

+0.5

0

-0.5

-1

6

Sources of perceived response burden

• Survey Design Factors – Characteristics of 6 web questionnaires directed to different industries– (The time it took to fill in the questionnaire)

• External conditions– Size: The number of employees– Information sources: Documentation problems that made the information

collection time consuming– Division of labour: The number of persons who took part in the response

process

• Personal Characteristics– Availability: How easy or difficult it was to set aside time for the

questionnaire– Motivation: How useful for the business and society the respondent felt that

statistics produced from the data were– Competence: If he/she was a first time or experienced respondent

7

2006 Web Response and PRB Response for Annual Business Surveys

Web response PRB response

Hotels and restaurants 31 % 43 %

Transport & communication

31 % 40 %

Sea transport 41 % 30 %

Service industries 33 % 45 %

Domestic trade 33 % 44 %

Construction 31 % 43 %

8

Perceptions of the Information Collection and Completion of the 2006 Questionnaires

Index Minutes

0,01

0,10

98

55

9

Sources of Time Consuming Preparations

Different sources

Need for help

Waiting for information

Many questionnaires

Other reasons

10

Sources of Response Burdens27,0 %29,1 %17,3 %16,5 %16,5 %0 %10 %20 %30 %40 %50 %60 %70 %80 %90 %100 %

Mismatch

Calculations

Response categories

No. of questions

Terms

Layout

Other

E-technical

E-usability

11

Information collection and Perceived response burden index: Industries and completion time

IC-index PRB - index

Construction -0.22 -0.17

Sea transport 0.04 (0.02)

Transport and communication 0.03 0.09

Hotels & restaurants -0.05 0.10

Domestic trade 0.01 0.10

Service industry 0.12 0.32

Completion time 0 – 25 minutes 0,32

26 – 50 minutes 0,11

51 minutes + -0,16

12

PRB-index: External conditions

PRB- index

Size of 0 – 9 employees 0,11

establishment 10 – 19 employees (0,13)

20 – 49 employees 0,06

50 or more employees 0,10

Documentation None 0,15

problems 1 (-0,14)

2 ((-0,23))

3 -0,45

No of 1 0,15

Respondents 2 or more ((-0,01))

13

PRB-index: Personal characteristics

PRB- index

Available time Very easy to find 0,68

Quite easy to find 0,41

Neither easy nor difficult 0,06

Quite difficult -0,09

Very difficult -0,37

Perception of Very useful ((0,22))

usefulness Quite useful 0,30

Neither useful nor useless 0,18

Quite useless 0,09

Very useless -0,11

Experience No 0,04

Yes 0,11

14

The effect of survey design, external conditions and personal characteristics on PRB

• Calculations and mismatch between questions and available information are general problems in business surveys

• Sufficient time available for the questionnaire is essential for the perceived response burden.

• The number of questions was probably the most important difference between questionnaires

• Poor layout and usability problems were often mentioned by those who felt that the questionnaire was very burdensome

• The quality of documentation systems is probably the most important difference between industries

• The perception of uselessness goes together with perceived response burden

15

Web Response Patterns for Construction and Service Industry Questionnaires

Construction

Service

16

Questionnaires corrected

Construction 80,9 %

Sea transport 53,3 %

Transport and communication 63,3 %

Hotels & restaurants 79,7 %

Domestic trade 70,1 %

Service industry 33,4 %

17

The impact of perceived response burden on data collection and data quality

• Generally in business surveys..– the data collection time is very long and– the number of questionnaires that are corrected is very high

• When there is a pronounced difference in perceived response burden between questionnaires, there also seems to be a difference in…

– How quickly the data are received– The number of questionnaires that need to be corrected

• Problems with collecting the information asked for may also have an effect on these quality indicators

18

Perceived response burden of the paper and web versions of the Annual Manufacturing Industry surveys (2004 and 2005) PRB-

index

0,01

-0,08

0,39

0,32

19

Sources of response burden in paper and web versions of the Manufacturing Industry Surveys

One business Multi business

Paper Web Diff Paper Web Diff

Mismatch 53 53 0 72 67 5

Calculations 18 39 -21 36 56 -20

Terms 24 29 -5 22 78 -56

No of questions 5 10 -5 21 44 -23

Response categories

14 18 -4 21 22 -1

Layout 1 4 -3 3 22 -20

E-technical/usability 0 6 -6 0 11 -11

Other 6 14 -8 5 6 -1

SUM 121 173 -52 180 306 -126

20

(The perceived response burden of web versus paper questionnaires)

• When the Manufacturing Industry Survey changed from paper to web…

– The perceived response burden apparently went down– While the number of problems reported by those who felt the

questionnaire burdensome seemed to go up

A quote for reflection: “It takes 100 men to do as much harm as a computer can make within 1/100 of a second”