the solution to online panel sampling

The Solution to Online Panel Sampling

Managing Within and Between Panel Variability in Online Samples ‐

In the past five years, increasing scrutiny has been directed towards online data quality issues brought on by panelists engaging in “satisficing” behaviors (i.e., doing the minimum to satisfy the survey’s requirements) and the “professional respondent.” Many researchers began to wonder about the online methodology and whether it truly was a viable means to collect accurate and inexpensive data. Because some researchers could not get findings to replicate from one point in time to another, or when using a different panel supplier as a sample source, many in the industry began to wonder whether they should be using online data collection at all.

The Problem

When online data collection was just beginning, many panel companies built large panels and touted panel sizes in the millions. However, roughly 80% of their surveys were being done by 20% of their panelists.

Then in 2006, Comscore Networks reported at the CASRO conference that less than 1% of the respondents participating in the largest online survey panels in the US were responsible for 34% of the completed interviews. This might have been somewhat of an overstatement, but what it did do, though, is to bring the discussion of the “professional” respondent to the fore. Other research done by panel companies found that as much as 75% of a panel’s members belonged to three or more panels (Comley, 2005). Researchers began to question the validity of relying on a much smaller group of online respondents than was previously thought to be completing surveys.

Characteristics of Professional Online Respondents

One of the main characteristics of professional respondents is that they are more likely to participate in surveys, which up until the debate about professional respondents, was considered a good thing – i.e., panel companies touted their cooperation rates. This by definition, though, makes it harder to limit their participation and hence their impact on the data.

Initially the response by many researchers was to put a number of data quality checks into place and to “clean” the data to eliminate professional respondents. However, some of the research‐on‐research findings, which will be discussed below, indicate there are interesting and useful differences about professional respondents demonstrating that they are not all “bad.” Finally, many researchers have begun to point out that because of their ubiquitous nature, to eliminate them, either through screening or by cleaning them out of the final data set, would be unfeasible, expensive and potentially misleading.

Copyright Mktg, Inc. 2009. All rights reserved.

Researchers are divided on whether professional respondents negatively impact the data. There are some that would argue that professional, or acculturated, panelists give more accurate ratings because of their experience with the process. For example, in a study by Coen, Lorch and Piekarski (2005) inexperienced respondents tended to be more positive toward concepts and to make higher ratings on purchase intent scales than frequent survey takers. The research study reporting this finding concluded that the more experienced respondents’ answers were more in line with reality.

Another study (De Wulf, 2007) found that those who belonged to multiple panels were more positive to the research process, more willing to complete additional surveys and viewed others in the panel process in a more favorable light. In the same study, the stereotypical view of professional respondents only completing surveys to obtain an incentive was shown to be less than true. Instead, the most common motives for being on multiple panels was “to learn new things” (31%) and “the desire to help others” (25%), not the commonly assumed “interest in receive a financial reward” which came in at a much lower 16%.

On the other hand, professional respondents are more prone towards certain behaviors, which in their extreme, are generally undesirable to those of us in the research industry. Satisficing, the practice of providing less thoughtful answers to simply complete a survey is often correlated with straight‐lining. Confronted with a battery of attributes about a brand the tendency is to always choose one particular scale point – for example, “Agree somewhat.” While these respondents could legitimately be relatively non‐committal about the brand, this is also a tell‐tale sign of inattentiveness.

Researchers can detect inattentiveness vs. ambivalence with trap questions embedded at different points in the attribute list. For example, by including a statement worded both ways such as “I usually choose brand over price” and “I usually choose price over brand,” an inattentive respondent may be caught. A more direct alternative is to simply include as an attribute: “For this item, please choose the “Disagree Strongly” scale point.

A comprehensive study, conducted with multiple sample sources by Gittelman and Trimarchi, presented at the CASRO conference in February 2009, found that all three undesirable behaviors (i.e., speeding, failure to follow instructions, and inconsistencies) were present among all the major US panels and the incidence of these behaviors varied markedly by panel. In their research, they included a number of different sample sources including the “River,” social networking site and point system sites, as well as 11 of the top US panels.

Gittelman et al. operationalized the professional respondents in three different ways:

1. Respondents who claimed membership in five or more panels 2. Respondents who reported that they took online surveys “practically every day” 3. Respondents who answered as an open‐end that they took over 30 online surveys in a month

Their findings were that 36% of the panel respondents were members of five or more panels, 25% took surveys practically every day, and 15% took over 30 online surveys in the past month. Depending upon


the above definition, the percentage of professional respondents in one of the 11 US panels ranged anywhere from 18% to 62%.

Given these findings, the incidence of undesirable behaviors in the online panel “universe” showed that:

• Twelve percent of the respondents failed to follow instructions, ranging from 8% to 15% depending upon the sample source.

• Inconsistent responses ranged from 3% to 10% among the respondents with 7% being the average.

• A predetermined 10% of all respondents were considered “speeders” with a high for one sample source of 14% and a low of 6% on another.

How Professional Respondents Impact the Data

In the section above, what professional respondents think and do is discussed, but more relevant to us as researchers is how they impact the data we collect. Since every panel supplier has them in their panel, to continue to use online sample as a data collection methodology, it is critical to understand how they affect our data quality. In a study by Casdas, Fine and Menictas (2006), major attitudinal differences were found in respondents who belonged to multiple panels while there were no notable demographic differences. Further support for this is found in the study by De Wulf (2007) mentioned earlier. Among the many demographic variables examined, the only significant difference was found in a higher propensity to be unemployed/looking for work.

In the Gittelman, et al. (2009) study, the various panels were profiled on educational distribution, presence of children under 18 in the household and employment distribution. (Note: For comparability, quotas were set across all panels on age, gender, income and ethnicity.) Other demographics such as education and employment, not held constant by quota, varied significantly and meaningfully. Variability was heightened by known differences in panel recruiting methods. Outside of these, De Wulf (2007) concluded on the issue of sourcing: “… multi‐method recruitment for building online panels is not a necessary condition for building a good quality panel. Most recruitment methods have shown to recruit quite similar profiles.”

The behavioral data collected in the Gittelman, et al. (2009) research, however, showed marked differences in buying behavior between sample sources. Data was collected on 37 different purchasing related measures and all panels were combined to form a “Grand Mean,” which might be loosely taken to represent the known online universe. They then computed the percentage of panels that differed significantly from this grand mean on all the 37 dimensions. Significant differences from the grand mean were found on all but one measure, the tendency to choose price over brand. In contrast, 37% of the panels differed in high tech purchases, and 32% in downloading music and using coupons. Looked at another way, nearly all panels diverged from the grand mean on at least one of the 27 measures; only two did not. Of those that differed, one panel diverged from the grand mean nearly half the time (48%).


Best practice recommendations

In conclusion, there are demographic differences between professional respondents and other panelists, consequently there are profound and significant differences in how they respond to both attitudinal ratings and in their stated behavioral data. In addition, there are very few panels that do not exhibit some difference from the grand mean in the Gittelman and Trimarchi research. Even among the few panels that do not differ from grand mean, every panel has professional respondents in them. So what is a researcher to do?

We recommend that rather than try to eliminate professional respondents and/or attempt to find the perfect panel, the best course of action would be accept the state of the online research world as it is and instead to work toward controlling the variability. That is, since the grand mean on all measures included in the Gittelman and Trimarchi research is known, it is possible to select a sample from a subset of panels so that it matches the grand mean on pertinent dimensions. Selecting samples like this over time means that online study samples can be monitored for consistency and through various techniques described below will be consistent and will not experience spurious data variations that have collected due to differences over time or from one to another panel supplier.

The Solution

This can readily be accomplished because, Mktg, Inc., has devised a method to select a subset of panels to blend sample which matches the grand mean on buying behavior, the segment considered most critical to most clients. That is, by optimizing the mix from a subset of panels (usually 2 to 3) they can repeatedly produce a consistent sample.

Note that this is not simply a mixture of panelists from different sources. Even such a mixture might be prone to drift over time since panel suppliers’ sourcing changes over time. Instead, they have designed a survey instrument that consists of 37 data points related to buying behavior. In order to produce a consistent sample blend, they have segmented the data set on these 37 measures. In this segmentation, they have uncovered four buying segments:


Now, armed with the four buying segments, it is possible to optimally blend a different number of panelists from any three given panels to find the optimum subset to reproduce this segmentation without having to use some panelists from all of the panels. Confining the blend to three panels, the optimal solution may be derived by a computer optimization routine. This optimal solution is shown below:

Optimum Average Expected (1 SE)

Panels

M8 24% 33% M17 26% 33% M12 50% 34%

Root Mean Square Error 0.40% 2.36% 8.31%

Shown is one optimal panel combination blend, with the proportion of the sample that should be drawn from each panel, and the Root Mean Squared Error (RMSE) associated with that optimization (0.40%). For comparison purposes, a one‐third/one‐third/one‐third split is shown as well. Note that by simply


mixing together three panels without the optimization, the blend results in a higher RMSE (2.36%). Finally the Expected column shows what the average RMSE would be for all possible combinations of three panels optimized. (8.31%)

However, it is possible that even though this is an “optimal blend” from a statistical standpoint, these panel suppliers might not be as easy to work with as others, or might have higher costs. Below are two other panel combination solutions, where the three panel companies are chosen, but then the blend is optimized within those three to reduce the RMSE.


Panels

M8 0% 33% M13 91% 33% M16 9% 34%



Panels

M10 8% 33% M13 66% 33% M16 27% 34%


It is interesting to note that in the first set, panel M8 would not be needed at all to reach the optimal solution. Another critical result in this is in the bottom optimization – simply selecting three panels and mixing them equally can have a detrimental effect, producing more error than if any random three were chosen (12.3% vs. 8.3%).

Given this multiple panel solution, which would guarantee a high degree of consistency in online sample pulls over time, also with a high level of precision, we propose that a strategy for “multiple panel/optimized mix methodology” be adopted for all online data collection. This methodology can control for the bias introduced by professional respondents as well as eliminating the inconsistencies seen in using one online panel supplier over time or by engaging different research firms using different panel suppliers. It is even superior to a supplier who buys panelists from multiple panel houses but who has not optimized the mix of panelists from each to a meaningful variable of interest ‐‐‐ buying behavior.

Moving forward, the Gittelman, et al. research is continuing. More panels have been added to the mix and panels completed before are being updated. Therefore, the grand mean or the “known online universe” for the US is being continually updated. It will be possible to keep both the segmentation current as the panel pool to choose from widens. And the optimization routines guarantee a consistent


set of panelists for every research project. As for the future, research is under way to create the same data base in forty global markets soon the same blending methods will be available from Australia to the United Kingdom.


the solution to online panel sampling

Documents