Deirdre Giesen & Joep BurgerQ-Conference, June 2-5 2014, Vienna

Response quality in the Structural

Business Survey questionnaire

Outline

• Why this research project? • Some background on SBS • Previous results and current step• Data used• Quality indicators• Results• Issues for discussion

Purpose research project

1. Develop quality indicators for quality “raw data” of business survey.

2. Explore how this quality is related to – characteristics response process– data collection design

Possible use of such indicators

– Allow quick response during data collection (also: direct feed back to respondents during completion of electronic questionnaire).

– To guide further research and adjustments in design data collection.

Structural Business Survey

– Annual survey– 2nd largest business survey at SN (about 70000

business units in sample)– Collects detailed specification of income and costs

of businesses– Most industries and size classes included– Used for European SBS and Dutch National

accounts– About 200 versions of the questionnaire: common

core part with size-class/industry specific specifications

– Redesign introduced in 20065

Our previous research (1/2)

– General quantitative indicators developed that can be used for all types of SBS:‐ Item response to 5 core variables‐ Set of consistency rules ‐ Use of “other costs” item

– Related these to time (before / after redesign), size of business, type of industry, timeliness response, mode, response burden

6

Our previous research (2/2)

– Overall quality of indicators was rather high– Not many systematic and large differences

related to background variables studied however:‐ Smaller businesses perform worse than larger

businesses‐ Some improvement on item response and

consistency after introduction of electronic questionnaires

‐ After redesign in question block on personnel one core variable clearly higher item response, other clearly worse.

7

Current step of research

– Focus on one unique questionnaire, which allows detailed analyses of the quality of cost specifications.

– Selection: general manufacturing questionnaire for size class 5 ( 20-49 employees).

– For this questionnaire: pilot study with introduction of electronic questionnaire in 2005, introduction of total redesign in 2006.

– Data for 2003-2007 analysed, about 1400 respondents each year.

8

Example of calculation in SBS qnr

9

Quality indicators

1) Quality of calculation (6 sets defined)2) Overall item response: fraction of subitems used 3) Item response on set of 11 core variables4) Fraction of total value specified for each core variable.

10

Quality of calculations

11

Overall item response (costs)

12

Item response core variables

13

Fraction of total value specified

14

– Demonstrates methodology to quickly assess quality of this type of raw data.

– Seems useful to test if and how indicators can be used for - individual feed back to respondents.- standard monitoring of data collection.

- Illustrates that also for “hard business data” questionnaire design affects response process.

- Limited use of indicators for evaluation of design characteristics without experimental data.

15

Conclusions

Questions, ideas?

– What aspects of quality in raw data in business surveys do you monitor and/or feed back to respondents?

16