the data they are a changin' - tns

May 27-30, 2014 Congress Plaza Hotel & Convention Center #MRMW

Organized by

Title Sponsor Gold Sponsors

Silver Sponsors

MRMW Chicago May 27-30, 2014 #MRMW

Workshop Sponsors

Association & Education Partners

Networking Reception

Sponsor

Event App

Partner

Media Partners


© TNS

© TNS May, 2014 © TNS May, 2014

The Data They are A Changin’ Examining the impact of: Data Sources Data Collection Devices Questionnaire Lengths

Kirk Ward EVP, Research Methods and Offer Innovation

MRMW May 29, 2014

© TNS

© TNS May, 2014

2

We are facing a crossroads in data collection.

• The dramatic increase in the use of smartphones

• Difficulty in recruiting and retaining respondents for online panels

We are going mobile

But are we delivering high quality data?

To help guide our clients through this transition, we need to gain an understanding of the impact of differing sample sources and data collection methodologies

The Business Issue

© TNS

© TNS May, 2014

1. Does data quality

differ by sample

source?

Research on Research – Questions to be answered

3

2. Does data quality

differ by data

collection device?

3. What is the

impact of

questionnaire length

on data an on

termination rates?

4. What is the

impact of optimizing

the programming for

smartphones?

What impact do sample sources and data collection methods have on survey results?

© TNS

© TNS May, 2014

Methodology

4

Data Collection Platform

• PC • Tablet • Smartphone

Questionnaire

• Long – 10 minute target • Short – 5 minute target • Optimized vs. Non-

optimized

Interviews were conducted in April, 2014

Sample Source

• Panel A • Panel B • Router • River sample • Google Consumer Survey

Sample Composition

• General population, ages 18-64

• Balanced by age, income, ethnicity, and geography

• Differential sampling using methods currently used for each sample source

© TNS

© TNS May, 2014

Questionnaire Design

5

• Various Category Usage and Ownership

questions

• Mobile phone

• Brand own

• Brand attributes

• Concept test

• Purchase interest

• Uniqueness

• Value

• Liking

• Relevance

• Excitement

• Understanding

• Believability

• Carbonated soft drinks (Coca-Cola, Pepsi,

and Sprite)

• Consumed in past week

• Attributes for each brand

• Demographics:

• Income

• Education

• Marital status

• Household size

• Postal code

The “Short” questionnaire featured a subset of these questions

© TNS

© TNS May, 2014

Topline Findings

6

Sample Source:

• River Samples produced significantly overstated answers to many questions – sometimes dramatically so.

• Panel A and Panel B produced similar answers and answer patterns to each question, with Panel B answers being consistently a bit higher in terms of scale usage (although these differences were not typically meaningful).

• The Router samples had results that were similar to the answers and answer patterns seen in the Panel A results

• The Google Consumer Survey samples produced patterns similar to those produced by Panel A but answers tended to be a bit lower in terms of scale usage than the other sample sources

Device:

• Results were generally similar across device type in the pattern of the answers and the answers themselves.

© TNS

© TNS May, 2014

Topline Findings

7

Questionnaire – Optimized vs Non-Optimized:

• The Optimized Questionnaire - featuring simpler, conversational language and “gamified” response scripts - produced essentially the same answers and answer pattern as did the Non Optimized Questionnaire.

• However: Incompletion rates were much higher for the Non-Optimized questionnaire

Questionnaire Length:

• The longer questionnaire produced dramatically higher drop out rates.

• Both the Optimized and Non-Optimized questionnaires produced similar interview lengths with the Optimized Questionnaire taking slightly longer for respondents to complete.

© TNS

© TNS May, 2014

Examples from the data

8

TNS Mobile RoR

© TNS May, 2014

PC Tablet Smart Phone PC Tablet

Smart Phone

9

0

10

20

30

40

50

60

70

80

90

100 %

Panel A Panel B River

Router GCS

Category Usage – Smoked a cigarette

Standard Optimized Lumi – LSR Lumi – GMI GCS

PC Tablet Smartphone PC Tablet Smartphone Smartphone Smartphone PC Tablet Smartphone

Long Short Long Short Long Short Long Short Long Short Long Short Long Short Long Short Short Short Short

Standard & Optimized: Which have you done in the past month?

TNS Mobile RoR

© TNS May, 2014


Smart Phone

10

0

10

20

30

40

50

60

70

80

90

100 %

Panel A Panel B River Router GCS

Category Usage – Bought motor oil for your car





TNS Mobile RoR

© TNS May, 2014


Smart Phone

11

0

10

20

30

40

50

60

70

80

90

100 %

Panel A Panel B River Router

Category Usage – Traveled on a commercial airline

Standard Optimized Lumi – LSR Lumi – GMI

PC Tablet Smartphone PC Tablet Smartphone Smartphone Smartphone

Long Short Long Short Long Short Long Short Long Short Long Short Long Short Long Short


TNS Mobile RoR

© TNS May, 2014


Smart Phone

12

0

10

20

30

40

50

60

70

80

90

100 %


Category Ownership – Cat




Standard & Optimized: Which items do you currently own?

TNS Mobile RoR

© TNS May, 2014


Smart Phone




13

0

10

20

30

40

50

60

70

80

90

100 %


Brand of Mobile Phone Own - Blackberry

Standard: Which brand(s) of mobile phones do you personally use?

Optimized: Which brand(s) of mobile phones do you use?

TNS Mobile RoR

© TNS May, 2014


Smart Phone

14

0.0

1.0

2.0

3.0

4.0

5.0

6.0

7.0

8.0

9.0

Mean


Concept Test Purchase Intent




Standard: Which statement best describes how likely you would be to buy this product if it were available in stores where you shop? Optimized: How likely would you be to buy this product?

TNS Mobile RoR

© TNS May, 2014


Smart Phone

15

0.0

1.0

2.0

3.0

4.0

5.0

6.0

7.0

8.0

9.0

Mean


Past Week Consumption – Coca-Cola

Standard : How many times have you consumed each of these carbonated soft drinks in the past week?

Optimized: In the past week, how many times have you consumed each…?




Tim

es P

er

week

TNS Mobile RoR

© TNS May, 2014


Smart Phone

16

0.0

1.0

2.0

3.0

4.0

5.0

6.0

7.0

8.0

9.0

Mean


Past Week Consumption – Pepsi

Standard : How many times have you consumed each of these carbonated soft drinks in the past week?

Optimized: In the past week, how many times have you consumed each…?




TNS Mobile RoR

© TNS May, 2014


Smart Phone

17

0.0

2.0

4.0

6.0

8.0

10.0

12.0

14.0

16.0

18.0

20.0

Minutes

Panel A Panel B River Router #REF! #REF!

Interview Length




Average Length Short: 5 min Long: 12 min

TNS Mobile RoR

© TNS May, 2014


Smart Phone

18

0.0

10.0

20.0

30.0

%


Term Rates




Average Length Short: 5 min Long: 12 min

© TNS

© TNS May, 2014

Implications

19

• River sampling requires some caution

- We need to investigate which sources of River Sampling produce the most reliable results

• Sample blending needs to be done with care

- Panel A and Panel B are similar but we need to ensure consistent levels of blending if we are comparing results (waves) over time

• Device agnostic surveys appear to be very feasible

- Great news – helps response rate

- Comparing results from mobile surveys to databases is possible

• Keep Mobile questionnaires short – under 6 minutes

- We should strive to reduce questionnaire length regardless of data collection method as it impacts completion rates

• Optimize your questionnaires

- It’s good research practice and it reduces early termination

Title Sponsor Gold Sponsors

Silver Sponsors


Workshop Sponsors

Association & Education Partners

Networking Reception

Sponsor

Event App

Partner

Media Partners


May 27-30, 2014 Congress Plaza Hotel & Convention Center #MRMW

Organized by

the data they are a changin' - tns

Marketing

tns tns

data quality

data collection methods

sample source panel

data collection methodologies

optimized questionnaire

panel b answers

similar answers