the concepts contained in this presentation are the property of e-rewards, inc. duplication or...

The concepts contained in this presentation are the property of e-Rewards, Inc.Duplication or dissemination of this information without the express writtenconsent of e-Rewards, Inc. is prohibited.

Faster, Cheaper, Better

November 7, 2005

Kurt Knapton, Executive Vice Presidente-Rewards Market Research

[email protected]

Online Research: A Focus on Quality in Realizing the Promise of the

Internet

2 © 2004 e-Rewards, Inc. All Rights Reserved.

e-Rewards Market Research is the online quality leaderwww.e-rewards.com/researchers

e-Rewards Market Research serves its clients with high-quality online sample from its B2C, B2B, and specialty panels.

e-Rewards Business™ PanelOver 1,000,000 members

40+ firmagraphic dimensions

C-Level Executives 7%

Professional/Managerial 72%

College Degree or More 72%

Post Graduate Study/Degree 32%

Specialty Business Panels

• CxO Panel

• IT Decision Maker Panel

• Business Owners Panel

• Physician Panel

• 6 Other Business Specialty Panels

e-Rewards Consumer™ PanelOver 2,00,000 members

300+ profile dimensions

Geographically balanced

Mean Age 44

Men 53%

Women 47%

Specialty Consumer Panels

• Ailments Panel

• Life Events Panel

• Affluent Panel

• Traveler Panel

• 8 Other Consumer Specialty Panels


Session Topic: Online Research Quality

The dramatic shift in recent years towards online research has been driven largely by the desire for reduced field time and cost savings.

But has the promise of “faster and cheaper” desensitized the research community to potential quality issues when deploying online methodologies?


Online Research

Question to Researchers:

“Which of the following describes Online Research versus other research modes?”

Faster (Not in Debate – A Key Strength)

Cheaper (Typically the Most Cost Effective)

Better (It Depends on How You Conduct it!)


What is Better About Online Research?


“What aspects of the Online Research mode are better?

The Survey Instrument• Preferred by Respondents (More

Convenient)• Reduces Interviewer Bias• Doesn’t “Call in the Middle of Dinner”

(Polite)• Media Richness (Graphics/Video/Audio)• Question-Level Validation (No Data Gaps)• Ease of Operation (Self-Paced,

Start/Stop)• Perceived as Anonymous (Candid

Honesty)• Quiet/Non-Intrusive Interview Context• Longer Attention Spans (vs. Phone)• Eliminates Data “Re-Keying” Error





Respondent Control• Targeted Population Frames• Respondent Screening• National / International Reach• Storage/Retrieval of Profile Data History• Past Participation Tracking• Automated “Time to Complete” Capture





Sample Quality (the key issue)

• Methodological Purity?• How Representative?• Respondent Validation--“Survey Gamers”?• Respondent Duplication?• Response Rates?• Overusage?

“Better” (or even “As Good”) depends on Sample Quality


What is the Impact of Low Sample Quality?

“Quality Research Requires Quality Respondents”

GARBAGE IN GARBAGE OUT

Results integrity is ultimately at stake. Of course, this is true across all research modes.


What Defines Online Research Sample Quality?

We Believe There are 5 Key Areas of Sample Quality

SAMPLE RECRUITMENT

SAMPLE SCREENING

SAMPLE COMPOSITION

SAMPLE MAINTENANCE

SAMPLE INCENTIVES


How Can Online Sample Quality be Measured?

We think in terms of a 15 Point Checklist

SAMPLE RECRUITMENT

SAMPLE SCREENING

1. Control of Sample Sources / Pre-Validation (Where it all starts!)

2. Recruitment Method (“Closed” vs. “Open” – Pros/Duplication)

3. Recruitment Mode Diversity (Mix of Online & Offline)

4. “Double Blind” Screening Technique (Enforced Internal Validity)

Source: http://company.e-rewards.com/15points


15 Point Checklist for Sample Quality (Continued)

5. Fraud Prevention/Detection (An Ounce of Prevention…)

6. Sample Verification (Multiple Checks)

7. Normalization (Example: Gender Composition is Telling)

8. Segmentation (Deeper Profiling = More Sample Control)

9. Participation Rules by Topic (Track and Enforce)

SAMPLE COMPOSITION


How Can Online Sample Quality be Measured?


What Defines Online Research Sample Quality?

10. Sample Profile Data Recency (Old Data is Less Reliable)

11. Survey Frequency Controls (Over-Surveying Hurts Everyone)

12. High Response Rates (Lessens Non-Response Bias Risk)

13. High Retention Rates (Key for Longitudinal Observations)

14. Respect for Respondent Privacy (Promotes Honesty & Trust)

15. A “Fair Value Exchange” with Respondents (Value Their Time)

SAMPLE MAINTENANCE

SAMPLE INCENTIVES


15 Point Checklist for Sample Quality (Continued)


Other Voices in the Industry are Speaking About Online Research Quality Concerns

comScore

In a recent study, comScore used observation-based research to track their members and quantify the concentration levels and activity of “professional survey respondents”.

Sigma Validation

Mary Beth Weber at the AMA 2005 Marketing Research Conference, Boston, MA September 25-28, 2005, “Why Validate Internet Research?”

20/20 Research-Online

September 1, 2004 White Paper by Rachael Krupek entitled: “Handling Paid Survey Sites.”

American Sports Data

Harvey Lauer, “You Say Evolution, I Say Devolution: Has Data Collection Improved or Gotten Worse?” Quirks’ Marketing Research Review July/August 2005


What comScore Sees:

“More Than 30% Of All Online Surveys Are Completed By Less Than 0.25% Of The Population

A recent study by comScore has confirmed the dawn of the "professional survey respondent," and validated the growing concern that such consumers do not represent the broader population. Further, panelists in this small group take an average of 80 surveys over a 90-day period — with some taking several surveys per day!

comScore research also shows that members of the panels offered by most of the leading online survey suppliers are, on average, members of as many as seven other panels! It goes without saying that these levels of saturation are unacceptable and can be expected to have a significant negative impact on the quality and accuracy of panelists' survey responses.

The industry is at a crossroad — market factors render a complete return to RDD impractical, but there is clearly a need to identify methods to improve the quality of online samples and associated responses.”Source: http://www.comscore.com/custom-research/sample.asp Also referenced by Mary Beth Weber, Sigma Validation at AMA 2005 Marketing Research Conference, Boston, MA September 25-28, 2005, “Why Validate Internet Research?”


What 20/20 Research–Online Says:

“Almost all paid survey sites encourage members to join all 250-450+ research companies’ panels on their list so their members can have the chance to participate in hundreds of surveys, focus groups and mystery shops.”

“Some sites even offer software to ‘help you fill out your surveys up to 300% faster.’ In essence, these sites teach people how to be professional respondents.”

Because these sites are legal as long as they deliver the list/report/database consumers are paying for, market research professionals have to be proactive in protecting the integrity of their databases by using some of the following practices:

…Decide if you will accept respondents from paid survey sites into your database at all or conditionally

…Contact paid survey sites and ask them to remove your company from their list/report/database”Source: September 1, 2004 White Paper by Rachael Krupek entitled “Handling Paid Survey Sites.” http://www.qualtalk.com/news/wp040901.htm Also referenced by Mary Beth Weber, Sigma Validation at AMA 2005 Marketing Research Conference, Boston, MA September 25-28, 2005, “Why Validate Internet Research?”


Why Should I Listen? Does it Matter?

We asked these questions ourselves.

As a result, we decided to enter into a 6-month observational test during the first half of 2004 to test out one of the 15 points of online sample quality differentiation:

Online Panel Recruitment Method(“Open” vs. “Closed” Approach)


What is Meant By “Open” vs “Closed” Online Panel Recruitment / Sample Sourcing?

“Open” Online Panel Recruitment may be defined as a method of allowing any person who has access to the Internet to “self select” and enroll into a market research panel.

By contrast, “Closed” or “By Invitation Only” Online Panel Recruitment may be defined as a method of inviting only pre-validated individuals or individuals who share known characteristics to enroll into a market research panel.


The “Open” Recruitment Problem: Self-Selecting Professional Survey Takers

The Open Door

Literally scores of pages of links to “open” online panel recruitment sites


“Open” Panel Recruitment Example #1



Dozens ofpanels sourcing members from the same place


“Open” Panel Recruitment Example #3 (Con’t.)

Over 300 more Panels Listed


CASE STUDY: A Comparison of “Open” vs. “By Invitation Only” (or “Closed”) Sample Sourcing

e-Rewards Market Research, an online sample provider based in Dallas, Texas, conducted a 6-month observational test during the first half of 2004 to compare and contrast “open” sample/respondent recruitment quality vs. e-Rewards’ controlled “by invitation only” or “closed” sample/respondent recruitment approach.

We wanted to know if we were “missing the bus” by not accepting members from paid survey sites and online panel aggregator sites.

The following pages present the key findings of our study.


CASE STUDY: Experiment Methodology Overview

Between January 2004 – June 2004, e-Rewards Market Research enrolled 38,162 panel members (representing less than 2% of all currently enrolled panel members) into its Consumer Panel using an “open” enrollment methodology.

In a parallel tracking experiment that was conducted for an additional year of study (July 2004 – July 2005), observations were made to compare these “open-sourced” members with e-Rewards’ “by invitation only” sourced panelists.

The following areas were compared for the two groups: Demographics, Motivations, Average Survey Complete Times, Fraud Detection, Response Rates, and Outside Panel Participation Duplication and Frequency.


CASE STUDY: Demographic Comparison

“Open” Sourced vs. “By Invitation Only”

Source: e-Rewards Market Research panel statistics, 2005Source: e-Rewards Market Research panel statistics, 2005


CASE STUDY: Demographic Comparison




CASE STUDY: Motivational Profile Comparison


2.19 Answers per Respondent 1.77 Answers per Respondent



CASE STUDY: Motivational Profile Comparison


Note: Panelists are asked during panel enrollment to provide a maximum number of survey e-mails preferred.



CASE STUDY: Professional Survey Taking Behavior Comparison: “Mean Survey Time”

Mean Time to Complete =8 minutes : 22 Seconds

14% Faster

Mean Time to Complete =9 minutes : 45 Seconds


Source: e-Rewards Market Research panel statistics, 2005



CASE STUDY: Professional Survey Taking Behavior Comparison: “Survey Time Outliers”


% of “Too Fast” Outliers Average per Study

2.1%

% of “Too Fast” Outliers Average per Study

0.6%



CASE STUDY: Response Rate Comparison

Average Response Rate =21.8%

Comparable

Average Response Rate =22.7%

Comparable




CASE STUDY: Outside Panel Membership (e.g. Cross Panel Duplication)




CASE STUDY: Outside Panel Survey Frequency(e.g. Cross Panel Duplication)




CASE STUDY: Outside Panel Survey Frequency




This is a Global Market Research Industry Issue

Market Research Trends Affecting our Industry in the Next 5-10 Years? 35% answered that “The Internet will continue to revolutionise the business”

Most Serious Threats to the Industry in the next 5-10 Years? 54% said, “Clients lacking skills to recognise the difference between good and poor quality of research by clients”

Key Factors for the Market Research Industry in the Future? 58% responded, “Standards of Performance/Quality Standards”

Source: CASRO’s 30th Annual Conference on September 28-30, 2005, Gunilla Broadbent (ESOMAR Council Member & Treasurer) presented World ESOMAR Research findings from Vision 2010 Survey


In Summary

The many inherent advantages of online research will likely continue to propel its adoption as a leading source of research responses.

However, the delivery of “faster and cheaper” research should not stop the research community from addressing online quality issues and establishing meaningful quality metrics.

The concepts contained in this presentation are the property of e-Rewards, Inc.Duplication or dissemination of this information without the express writtenconsent of e-Rewards, Inc. is prohibited.

QUESTIONS AND ANSWERS

THANK YOU!

November 7, 2004

Kurt Knapton, Executive Vice Presidente-Rewards Market Research

[email protected]

the concepts contained in this presentation are the property of e-rewards, inc. duplication or...

Documents