We are proud to issue the second ORQC Newsletter to inform the industry on

progress being made against the issues and opportunities addressed by the ARF’s Online Research Quality Council. As a reminder, we originally formed the ORQC in order to create a positive and constructive forum to bring together the leading buyers and sellers of online research to effectively address the key quality-related issues and opportunities being faced by the market research indus-try. Since the official launch of the council, we have achieved active participation across the industry with 20 client companies, more than 20 research agencies, and seven other professional associations, all working toward a common goal.

Taking the helm of the ARF’s Online Re-search Quality Council (ORQC) at a new confluence of insights, solutions, and metrics is Stan Sthanunathan, Vice President of Mar-keting Strategy & Insights at Coca-Cola. Stan assumed this key role on June 9th, 2009 - an opportune time as definitive results from the ARF-led Foundations of Quality (FoQ) re-search-on-research project became available.

Stan, a proponent of an action agenda, says: “Having lived through quality chal-lenges, I have a very high level of passion for this subject. I firmly believe that our work will

impact the industry in a big way. The need of the hour is an industry solution. But, we do not have the luxury of waiting for a long time to come up with a perfect solution. Let us set our respective agendas aside and work together to resolve this challenge.”

Robert Tomei, Founder and Co-Chair of the ORQC adds: “Since the very beginning of the ORQC, Stan has been a tremendous advocate of developing a solution based on the facts, and now that we have achieved that with the ‘Foundation of Quality’ study, we can move into identifying solutions that the Industry can adopt to address the real is-sues around online research quality. Having the right leaders and companies represented on the ORQC will ensure that we achieve this objective”.

Stan has instilled a sense of urgency that respects the hard work of the ARF and indus-try leaders who are contributing their time. This collaborative commitment to a “90-Day plan” will culminate in a presentation to the industry on September 29 of a Version 1.0 Industry Solution for Online Data Quality. With the cooperation of other industry as-sociations, the ARF will ultimately provide a program with recommendations regarding metrics and business practices, templates, definitions, and training that will enable buy-

ers and sellers to work together to bring the industry to a better place in terms of data quality, comparability, and the trustworthi-ness of online research results.

Upd

ate

Advertising Research Foundation’s Online Research Quality Council

NEW LEADERSHIPStan Sthanunathan of Coca Cola Takes ChargeFindings from the FoQ study are already paying dividends, leading to a better understanding of ways to boost data qual-ity and the improvement of online survey response quality, while enhancing the reliability of results.

Online Research Quality

Stan Sthanunathan Co-Chair, Online Research Quality CouncilVP, Global Knowledge & Insights, Coca Cola

“Stan has been a tremendous advocate of developing a solution based on the facts ...”Robert Tomei – IRI & Co-Chair, ORQC

In today’s world of swift and rapid change, knowledge is power. Just a few short weeks

after the presentation of the preliminary results of the FoQ study at the ARF’s 2009 Annual Re:Think Conference, topical papers based on the FoQ results were created by Bob Walker, President of Surveys & Forecasts, LLC, Joel Rubinson, Chief Research Officer and Dr. Raymond Pettit, Senior Vice President at the ARF. Following the careful participation and vetting processes established at the out-set of this project, the briefs were reviewed by the ORQC leadership, committees, and supporters of the project.

The ARF is now issuing three papers based on the FoQ results: (1). The Impact of Overlap, Multi Panel Membership, and Duplication on Online Data Quality; (2). The Response Quality of Online Surveys; and (3). The Consistency, Reliability, and Inter Study Comparability of Online Research Results.

The Knowledge Briefs are available to ARF Members on MYARF and to non-members by request and for a fee.

The Foundations of Quality study collect-ed over 100,000 completed survey responses from 17 panels in a rigorously designed re-search project. The question with the most important implications for the industry was: Why do survey results not replicate? Pan-els appear to produce different answers, so switching panels or changing the mix of pan-els can lead to results that are not compara-ble across studies. Also, panel practices vary widely, which influences results; for example, in the FoQ study it was found that purchase intent is higher in panels that offer cash incen-tives. Such panels also seem to attract more panelists who say they are motivated to take surveys for money.

Dr. Tom Evans of ESPN offers encourage-ment about the next steps of the project: “As

the FoQ initiative moves into the next phase, ESPN, as users of online research, is excited to see that the ORQC has committed to mov-ing quickly to address the original study’s re-sults and how they impact what we think we know and what we do not know about online data quality.”

The Online Research Quality Council’s newly formed Industry Solutions group,

led by Kim Dedeker, Kantar; Mark Berry, Syn-ovate; and Tomas Emmers, Unilever has cre-ated three focus committees tasked to work on the primary quality issues confronting online panel research. Based on the results of the ARF’s Foundations of Quality research study, extensive secondary research, and an inspection of best practices in the industry, these subcommittees are tackling the impor-tant job of turning results into actions and guidelines.

John Willard, Bayer and Jissoon Barton, Synovate will co-chair a group that will look at online panel management issues with an eye to improving consistency and reliability. Justin Fromm, ABC and Deb Santus, TNS will

lead a group focused on finding better ways to manage survey delivery. Simon Hough, Microsoft and Ashley Grace and Jeff Cox, ARS Group will be exploring ways to improve response quality and other related issues, such as managing survey length.

In tandem, a Metrics Committee, led by Jonathan Jephcott, Synovate, Stephen Oi,

Kraft, and Steve Segur, TNS will be coordi-nating and integrating their work on the de-velopment of key performance metrics and guidelines with the new committees.

The groundbreaking work of the Industry Solutions group will be blended into a Ver-sion 1.0 Industry Solution presentation to the industry on September 29.

RESULTSFoQ Project Begins To Bear FruitThe ARF’s Foundations of Quality (FoQ) project begins to bear fruit with the creation of ARF Knowledge Briefs: executive reports based on the FoQ research-on-research results.

WORK IN PROGRESSThe Road to Version 1.0The Foundations of Quality Research-on-Research Project has produced a definitive report on online research quality improvement. Results from this study offer findings and guidelines on measuring and assessing the quality of online research data.

FoQ Knowledge Briefs are available

free to ARF members on My ARF and to

non-members for $399.

(For questions, please contact Raymond

Pettit, SVP, Research and Standards at

rpettit@thearf.org).

Download or purchase at my.thearf.org.

On Tuesday, June 9th the ARF Online Research Quality Council presented

detailed findings from an unprecedented US R&D project regarding online data qual-ity, called “Foundations of Quality” (FoQ). Beyond the fact that it was about $1MM in research, it was unprecedented and quite remarkable as 17 leading online panel com-panies cooperated with each other and with large buyers of marketing research in a col-laborative and transparent way. We studied 700,000 panelists and over 100,000 survey responses from 17 panels in a rigorously de-signed research project.

Competitors and trading partners were drawn together at the ARF by a sense of ur-gency to reestablish the trustworthiness of online research ... by our shared future.

Let me discuss findings against four of the big issues that were on the table.

1 Some high profile examples have been reported of study results not replicating. Why is that?FoQ data prove that, for each of the 17 pan-els, results replicate within panel (within the limits of sampling variation) but do not nec-essarily replicate across panels. This means that buyers will need to be cautious about switching suppliers when data comparabil-ity to other study results is a main consider-ation. Furthermore, because suppliers often draw on sample sources beyond their own panels, buyers must engage in conversations with suppliers about any change in sample source.

We found two reasons worth noting for lack of comparability but not powerful enough to fully explain differences across panels:

• Panelist longevity: newer recruits are more likely to give positive purchase intent towards concepts. The longev-ity profile differed dramatically across panels.

• Those motivated by monetary rewards to take a survey appear to be more posi-tive to concepts and some panels stress such rewards while others do not.

2 Some feared that there is a small group of “professional respondents” on everyone’s panel, doing it for the money, and gaming the system rather than providing thoughtful answersFoQ proves that this is not true. Overlap is less than thought. Over 80% of e-mail addresses appear on only one panel and the collective pool of unique e-mails in the US is estimated at something over 5.5 million. For historical comparison, this is probably 2-4 times greater than the pool of mail panelists that NFO, HTI, and Market Facts collectively had in the 80s-90s. Also, mostly people are motivated to join online research panels by a desire to share their opinions, rather than for the incentives. Perhaps the most telltale finding is that those who take MORE surveys per month (up to 10) provide MORE thoughtful answers (i.e. less likely to straightline their answers or fail trap questions.) In other words, if anything, being “professional” is a GOOD thing as it relates to respondent engagement with the activity of survey-taking.

In fact, as it turns out, the biggest cause of not providing thoughtful answers is long surveys!

3 Are people taking the same survey more than once?On studies where qualification criteria are not very restrictive, “duplication” is mostly a sin-gle digit issue (depends on the panel or pair of panels) but if the qualification criteria are more specialized, it can more prevalent and

damaging. The industry must adopt practices that will address this.

4 Are things going to get better in the near future?Yes! We all feel the sense of urgency so The ARF and industry leaders who have contrib-uted their time and expertise will continue; we are committing to a 90 day plan. With the cooperation of other industry associations, we will provide a program with recommen-dations regarding metrics and business prac-tices, templates, definitions, and training that will enable buyers and sellers to work togeth-er to bring the industry to a better place in terms of data quality, comparability, and the trustworthiness of online research results.

FOUR ISSUES ON THE TABLEOur Shared Future Regarding Online Data QualityPosted June 10, 2009 in CRO-ing About Research, ARF Chief Research Officer Joel Rubinson’s Blog. blog.joelrubinson.net

Joel Rubinson Chief Research Officer The Adversing Research Foundation

“Competitors and trading partners were drawn together at the ARF by a sense of urgency to reestablish the trustworthiness of online research...”

READ & DISCUSS: blog.joelrubinson.net

An ORQC Special MeetingPresentation of the FoQ Version 1.0 Industry Solutions Plan

Tuesday, September 29 • 2:00–4:00pm1 Time Warner Center (58th St., between 8th and 9th Avenues)Screening Room, 10th FloorNew York, NY 10019

A very important meeting is set for Tuesday, September 29th, 2009, from 2-4 PM, at the Time Warner Center’s Screening Room. The culmination of nearly

two years of work on the Foundations of Quality initiative will be presented to the industry. This meeting will be open to ARF members and non-members, but due to space limitations we will be requiring registration. Register at my.thearf.org.

SAVE THE DATE – SEPTEMBER EVENTThe Foundations of Quality Version 1.0 Industry Solutions Plan

ORQC Governing BoardCO-CHAIR Stan Sthanunathan VP, Global Knowledge & Insights Coca Cola CO-CHAIR Robert Tomei President, Consumer & Shopper Insights Information Resources, Inc.John Willard Director, Market Research & Consumer Relations Bayer Consumer CareJames Mendelsohn Vice President, Corporate Strategy Capital OneBryan Mahlmeister Global Consumer & Product Research; Senior Manager, Study Design & Fieldwork General MotorsSteve Oi Sr. Consumer Insights Manager KraftKim Dedeker Chair, North America KantarEfrain Ribiero COO Lightspeed ResearchSteven Schwartz Group Vendor and Quality Manager, Central Marketing Research and Insights MicrosoftSteve Coffey Chief Research Officer NPDBob Walker Principal Surveys & ForecastsSteve Segur Vice President, Panel Development & Research TNSTomas Emmers Consumer & Market Insights Director Unilever AmericasTom Evans VP, Audio Research and Special Projects ESPNRenee Smith VP Global Operations Ipsos Mark Berry EVP Consumer & Business Insights Synovate Jonathan Jephcott Executive Vice President Synovate

Industry Solutions CommitteeCO-CHAIR Kim Dedeker Chair - North America KantarCO-CHAIR Mark Berry EVP Consumer & Business Insights SynovateCO-CHAIR Tomas Emmers Consumer & Market Insights Director UnileverAshley Grace President, Strategy and Partnerships ARS GroupBirgi Martin Advertising Research Manager GoogleDebra Santus Senior VP, IIS NA Access Panel IpsosDon Gloeckler Manager, Next Gen Research Procter & GambleDyna Boen Vice President, TrueSample MarketTools, Inc.Jackie Lorch VP, Global Knowledge Management SSIJames Mendelsohn Capital One Capital OneJeff Cox CEO ARS GroupJisoon Barton Senior Vice President SynovateJoe Giacobbe SVP, Operations & Services Vision CriticalJohn Ouren EVP/GM - Zoom Business Market ToolsJohn Willard Director, Market Research & Consumer Relations Bayer - Consumer CareJonathan Jephcott Executive Vice President SynovateKate Whittington Group Manager, Guest Insights TargetKevin Umeh CEO CintMainak Mazumdar SVP, Global Measurement Science/Emerging Media NielsenMarjette Stark Matt Dusig CEO / co-founder United SampleSimon Hough Senior Advertising Research Manager MicrosoftSimon Kooyman CEO Knowledge NetworksSusan Frede Vice President Research & Panel Management TNSSteven Oi Sr. Consumer Insights Manager KraftSteve Segur VP, Panel Development & Research TNSTed Christenson Consumer Insights Manager General MillsThomas Evans Vice President of Audio Research/Special Projects ESPN