the impact of response format on relations among intentions, attitudes, and social norms

Marketing Letters 1:2, (1989): 161-170 0 1990 Kluwer Academic Publishers, Manufactured in the Netherlands.

The Impact of Response Format on Relations among Intentions, Attitudes, and Social Norms PAM SCHOLDER ELLEN Georgia State University

THOMAS J. MADDEN University of South Carolina

Key words: Response Consistency Error, Theory of Reasoned Action, Attitude-Intention Consis- tency, Response Bias

February 9, 1990

Abstract

This research investigates the response consistency error which may be induced by certain com- mon methodological practices in tests of predictive models such as the theory of reasoned action (Ajzen and Fishbein 1980). Specifically, the physical presence of a questionnaire was expected to provide respondents an inducement and basis for maintaining consistent responses. In addition, the practice of presenting all measures of a given construct together seemed likely to heighten correlations among constructs, resulting in heightened predictions. Two experiments were conducted in which respondents completed measures of the theory of reasoned action for a variety of behaviors using either a paper or computer-administered questionnaire with items in standard or random order. The results of these studies indicate that, contrary to expectations, the degree of attitude-intention consistency across behaviors may be attentuated in standard paper administra- tions.

While researchers have long recognized that respondents’ ratings of stimuli may contain systematic sources of error (Cooper 1981, Thorndike 1920, Wells 1907), several authors have recently focused on the effects of the measurement process or context on the processes by which respondents generate answers to surveys (Feldman and Lynch 1988, Tourangeau and Rasinski 1988). Feldman and Lynch (1988) argue that the measurement process itself shapes “the nature of the com- putational and retrieval processes by which answers to survey questions are generated . . . [and] can increase or decrease revealed correlations among these constructs” (p. 424). They further suggest that the questionnaire items themselves may be used to “create” consistent responses. They call this effect “self- generated validity”-and suggest that even the ordering of the questions (whether A precedes B or B precedes A) can create different results.

One area in which concern with such response bias has been raised in the past is in tests of the association among constructs of intention-behavior models such as the theory of reasoned action (Beckwith and Lehmann 1975, Beckwith, Kas- sarjian and Lehmann 1978, Feldman and Lynch 1988, Laroche 1978). The physical

162 PAM SCHOLDER ELLEN AND THOMAS J. MADDEN

presence of a questionnaire allows the respondent to refer back to previous answers in formulating responses to new questions or to look ahead to upcoming questions. There is a physical reminder of the degree of the rating on previous answers and, according to Tourangeau and Rasinski (1988), a basis for developing an interpretive framework by which to construct appropriate responses. When related items are physically presented together, previously retrieved (or com- puted) information is available in working memory, making it available for use in subsequent judgments. These measurement practices would seem likely to heighten respondents desire to maintain consistency as well as making it physically possible to do so.

The purpose of the studies described here was to measure the extent to which a standard paper questionnaire administration procedure affects correlations among the constructs in tests of the theory of reasoned action. To assess the effects of the presentation method, subjects were administered the same set of items using either a paper-and-pencil questionnaire or a computer-administered questionnaire (CAQ) (Liefeld 1988).’ The relevant benefit of the CAQ program was the ability to present one question at a time. Respondents answered each question before the program advanced to the next and had no means of referring back to previous answers or looking ahead to upcoming questions.

1. Experiment 1

An experiment was designed to test the extent to which different measurement formats affected the relationships among the constructs specified by the theory of reasoned action. Subjects completed either a paper questionnaire (PAPER) or a computer-administered questionnaire (CAQ). The paper version and a computer version were presented in standard order (STANDARD); that is, all measures for each of twelve behaviors were presented together. As an initial test of the effect of order of questions, a CAQ version was also presented in random order. The random order (RANDOM) survey presented items in random order across constructs and behaviors.

It was expected that these procedures would differentially affect the observed relationships between the constructs of the theory of reasoned action. Specifi- cally, it was expected that the PAPER method would result in greater consistency than the CAQ method because respondents would be forced by the latter method to consider only one question at a time. A randomly ordered presentation was expected to reduce the individual’s ability to remember previous answers and therefore result in lower correlations among items.

Focus group interviews with student subjects were conducted to generate a variety of behaviors that might be performed within a typical three-week summer period. Twelve behaviors identified by the focus groups were “see a movie at a theater,” “ go swimming, ” “rent a movie on videocassette,” “eat at an expensive restaurant, ” “play tennis/golf,” “attend classes,” “be on time for class,” “go to

RESPONSE FORMAT 163

a nightclub with friends,” “go to a concert,” “go to a special event park,” “ix- ercise,” and “go to a party/cookout .”

For each behavior, respondents were asked their attitude, subjective norm, and intention to perform the behavior. Each construct was measured with two items (c.f., Ajzen and Fishbein 1980) and operationalized as the sum of the responses to the items. A sample of 112 undergraduate students were assigned to either a paper version (n = 70) or a computer-aided version (n = 50) of the questionnaire. The paper questionnaire presented all questions for each behavior together and on one page (PAPER/STANDARD). For half of those using the computerized version, the ordering of the questions was identical to the paper version (CAQ/ STANDARD). The remaining subjects received a randomly ordered version of the questionnaire (CAQ/RANDOM).

1 .l. Results

Prior to examining the relations among constructs, means for each construct were examined to assure that differences did not exist between administration forms. (See table 1 for means.) Mean differences could indicate that certain formats may cause polarization of responses (.Tesser 1978). A oneway ANOVA indicated no statistically significant differences among the means for attitude (F = .75, pp = .62), subjective norm (F = 2.6, p = .07) or behavioral intentions (F = 2.2, p = .I 1) across formats.

It was predicted that the standard paper questionnaire would result in higher correlations among constructs and consequently better prediction of behavioral intentions. Behavioral intentions were regressed on both attitudes and subjective norms for each format. Table 2 presents the adjusted R’ for each of the 12 behaviors by response format and the average R2 across behaviors. The results do not support expectations; in fact, the results were opposite the predictions. There was no strengthening of predictive relations. Across all behaviors, the average R* was .40, .63 and .69 for the PAPER/STANDARD, CAQSTANDARD, and CAQ/RAN- DOM versions, respectively. For all 12 behaviors, the explained variance was greater for the computerized versions than for the paper method. Interestingly, for 7 out of the 12 behaviors, the CAQ/RANDOM version had the highest R2. This format should have offered the least likelihood of a respondent maintaining consistency.

Previous authors (Beckwith and Kibilius 1978, Beckwith and Lehmann 1975, Wilkie, McCann and Reibstein 1973) have suggested that within-subject analysis is more appropriate for examining issues of response consistency. For this reason, the data were also analyzed within-subject. As predicted, the correlations between the items measuring the same construct were lowest for the CAQIRAN- DOM method in all but one case. (See table 1.) However, as in the previous analysis, the correlations among the constructs and the R? for the model are highest for the CAQ/RANDOM format.


Table 1. Average within subject correlations and means by format

Correlations Means

Format- && SN,SN, I,I, ATT+BI SN+BI R* ATT SN BI

Paper/Standard .65 .I2 .90 .57 .58 .61 9.86 11.63 11.02 CAQ/Standard .39 .65 .90 .73 .55 .66 10.32 11.44 11.07 CAQlRandom .63 .56 .84 .75 .67 .I3 10.40 11.60 10.50

Table 2. Comparison of adjusted R* across presentation formats

Behavior Paper/Standard CAQlStandard CAQlRandom

Going to movie .46 .64 .61 Going swimming .45 .72 .67 Renting videocassette .64 .67 .89 Going to expensive restaurant .24 .70 .45 Playing tennis/racquetball .51 .90 .89 Attending all classes .22 .28 .73 Being on time to classes .16 .27 .63 Going to nightclub .69 .I5 37 Going to concert .43 .54 .56 Going to special event park .43 .67 .44 Exercising .24 .91 .99 Going to a party .38 .53 .68

Average R2 .40 .63 .69

One possible explanation for this finding is that the subjects using the computer were more involved with the task and therefore were more thoughtful in supplying answers. To assess this notion, a second experiment was designed to measure subjects’ involvement with the task and to look at format effects for a larger number of measures per construct.

2. Experiment 2

The first study was designed to look at effects across a number of behaviors. For this reason, only two global measures of each construct were used. Experiment two was conducted to extend the results of the first experiment using a larger number of measures for a limited set of behaviors. In this second study, the de- terminants of attitudes (i.e., the beliefs and evaluations) were included as well as global measures of attitude (Ajzen and Fishbein 1980).2 Measuring the determi- nants of attitudes was expected to increase consistency for the STANDARD order and decrease consistency for the RANDOM version.

RESPONSE FORMAT 165

Only two behaviors were examined in this study: exercising regularly in a three- week period and donating blood. Exercise was chosen because of the stark difference in the R2s found in study 1 (see table 2) while donating blood was selected because of its frequent use in testing the theory of reasoned action (Sheppard, Hartwick, and Warshaw 1988). The same three questionnaire presentation formats used in experiment one were used here. In addition, this experiment incor- porated a paper version (PAPER/RANDOM) that mirrored the computer-aided random format.

2.1. Measurement

Focus group interviews and previous research (Ajzen and Fishbein 1980, Bagozzi 1982) were used to determine the modal salient beliefs for the two behaviors. Eleven beliefs were used for donating blood and fifteen beliefs for exercising.3 The determinant measures of attitudes (Xbiei) were operationalized as the sum- mation of the product of the beliefs and evaluations of the consequences. The global measure of attitude was the sum of five semantic differential items while behavioral intentions were measured with three items (cf., Ajzen and Fishbein 1980). After completing all other items, subjects’ involvement with the task was assessed with two seven-point Likert-type items: “I carefully considered my answer to each question” and “Giving an accurate answer to the previous questions was not at all important to me.”

Respondents (n = 264) were assigned to one of the four forms of the questionnaire. In the standard formats, respondents answered the items tapping beliefs, evaluations, attitude, and behavioral intentions for donating blood followed by the same sequence of questions for exercise. In the random versions, all questions were randomized. Cell sizes for the four treatments ranged from 55 to 70.

2.2. Results

Mean differences for each construct were first examined (table 3). No significant differences were found for exercise for Xbiei, attitude or behavioral intentions. However, for blood donation, significant differences were found for Zbiei (F = 3.67), p < .Ol) and attitude (F = 3.24, p = .03). Pairwise comparisons for Zbiei indicated significant differences between the PAPER/RANDOM version and the two STANDARD versions. For attitude, the significant differences were between both RANDOM versions and CAQ/Standard. No mean differences were found for behavioral intentions.

We then examined the correlations among the three measures. As shown in table 3, the correlations between the constructs for exercise were statistically significantly higher in all cases for the three alternative formats, compared to the standard format. In fact, the correlations between the attitude measures and be-


havioral intentions for exercising were twice as high for the alternative formats as the PAPER/STANDARD. Hence, similar to the results of experiment one, there appears to be an attentuation of the correlation when the method of questioning involves a paper questionnaire with constructs measured in sequential order.

For donating blood, the correlations between formats were not statistically different for any of the attitude-intention relationships. Thus, the effect of response format appears to vary across behaviors. Potential explanations for these findings are discussed subsequently.

The correlational differences between the behaviors may be attributable to the nature of the memory structures for each behavior. Judgments of certain behaviors may be more well-developed and more accessible than others. A number of authors (Chan et al. 1986; Fazio, Lenn and Effrein 1984) suggest that differences in accessibility may be indicated by response latencies or the amount of time required to generate an answer. The computerized formats allowed for the collec- tion of response latencies for each question. The average latencies for beliefs, evaluations, the global measure of attitudes and behavioral intentions were analyzed for the RANDOM version to assess any differences between the behaviors of donating blood and exercising for the RANDOM version. (No comparison to the CAQSTANDARD version was viable since latencies between the behaviors probably reflected learning effects because of the order of presentation of the behaviors.) A r-test on the mean latency for each measure indicated that the latencies for donating blood were significantly greater than those for exercising for all measures, except for behavioral intentions. These results would suggest that answers for donating blood were less easily retrieved or were constructed while those for exercising were more accessible. These differences in accessibility may account, to some extent, for the format effects found previously and consequently lead to speculation about the conditions under which these effects occur.

The longer time associated with blood may explain to some extent the mean differences observed for the attitude measures. Tesser (1978) showed that the greater the amount of time allotted to think about responses/feelings, the greater the polarity in attitudes. If, as conjectured earlier, the RANDOM format requires more effort, the increased time to generate responses may result in more polarized attitude measures. The means for the Zbiei measures in table 3 may reflect such time differences.

One objective of experiment two was to assess subjects’ involvement or interest in the task. It was speculated that when the subjects used the computer, they were more involved and consequently paid more attention to the answers they pro- vided, which might explain the attenuated R2s for the paper method in experiment one. The two involvement items were summed to form a composite score and differences in involvement by method and order were assessed with analysis of variance. The results indicated a significant effect only for the method of administration (F = 20.22, p < .Ol). Those subjects using the computerized method reported significantly more involvement (mean = 12.10)‘than those subjects using the paper format (mean = 10.89). This is consistent with O’Brien and Dugdale’s

RESPONSE FORMAT 167

Table 3. Comparison of means and correlations across presentation formats

Pearson R Means

Bbe+ATT Zbe+BI ATT+BI Zbe ATT BI

DONATING BLOOD Paper/Standard CAQlStandard Paper/Random CAQ/Random

EXERCISING Paper/Standard CAQlStandard Paper/Random CAQiRandom

.48 .48 .40

.42 .39 .51

.61 .44 s2

..58 .49 .36

so .64 .I1 .I9

.20 .19 169.48 45.68 16.11

.48 .Sl 191.43 45.69 17.16

.51 .63 176.44 44.18 16.19

.67 .75 189.39 45.48 15.58

11.04 36.41 11.71 15.83 37.66 11.90

- 3.81 34.41’ 9.97 2.80’ 36.122 9.85

ISignificantly different from both Standard versions 2Significantly different from CAQiStandard version

(1978) findings that computer-administration may be more involving and less mo- notonous than paper questionnaires. The higher correlations and better prediction by the model in the CAQ formats may to some degree be a function of the greater involvement with the computer. O’Brien and Dugdale (1978) report a higher standard deviation with CAQ respondents. They attribute the difference to greater attention to responses and a lower likelihood to use noncommital or middle response categories.

3. Conclusions

The results of the two studies indicate that a traditional paper response format may in fact attenuate the observed relationships among constructs in intention- behavior models relative to some other formats. The computer-administered format served in these studies as a basis for comparison and is not necessarily advocated as a better format for questionnaire presentation.

In the first study, the predictive value of the theory of reasoned action model was affected by the order of question presentation (i.e., standard vs. random) and method (i.e., paper vs. CAQ) for some behaviors but not others. Specifically, the adjusted R’ across all 12 behaviors was higher for the computerized versions and the differences for some behaviors were quite dramatic. The second experiment considered two behaviors in greater detail and found that the attitude-intentions consistency varied by method depending on the behavior. For donating blood, there were no significant effects of format on correlations between the constructs. However, for exercise, the correlations were greatest for the random computerized version and least for the standard paper. Hence, contrary to criticisms raised


about measurement errors in many intention-behavior models, the results of both studies suggest that relative to the alternative formats used, the standard order paper format may, in fact, result in lower observed correlations among constructs. Examination of the latencies for the random computerized version suggests that exercise may be more easily retrieved and that observed correlations are more affected by format and order for such behaviors. These results also suggest that responses to donating blood may have been more difficult to retrieve or were constructed and thus were not affected by the method of administration to the same degree.

One explanation for observed differences between the paper and computerized formats is the level of involvement by the subjects. Those using the computer reported that greater attention and care was taken in generating answers. The level of involvement or required effort also affected the format effects across behaviors. For behaviors requiring less effort to respond (e.g., exercising), the computerized and random methods may increase involvement. The correlations across construct measures then reflect this “noise.” For behaviors requiring greater effort (e.g., donating blood), the content appears to dominate and the administration format has little effect. As Wilson et al. (1989) suggest, greater observed consistency between attitudes and behavior may occur when high levels of attitude-behavior consistency did not exist prior to measurement. Future research can address these issues by directly manipulating the level of involvement or effort and the degree of pre-existing attitude-behavior consistency.

One difficulty with this type of research is the inability to determine which correlation or predictive model is the true one. It may be that for behaviors with more well developed and easily retrieved constructs (i.e., exercise vs. donating blood), the standard paper method may understate the attitude-intention relationships (e.g., .20 < .48) while the random computer method may overstate it (e.g., Sl > .44). The high observed attitude-intention correlation in the CAQIRAN- DOM format may also reflect “halo error.” The heightened difficulty of answering questions in this format may have encouraged respondents to rely on an overall attitude or affect in generating responses while the other versions may have facil- itated the retrieval process by the physical presence of previous answers and the sequential nature of the presentation order.

The results in these two studies were contrary to expectations. By replicating these patterns across the formats, we find support for the existence of some measurement and context effects other than those previously addressed. The somewhat inconsistent patterns across behaviors and methods would be considered undesirable and troublesome by many researchers. However, Tourangeau and Rasinski (1988) argue that these “unreliable” effects of measurement processes are, in and of themselves, interesting phenomena. By focusing on such measurement context effects, we can derive greater insights into the interpretative, retrieval, and judgmental processes of survey respondents. Future research will need to consider some of the issues and alternative explanations raised by the interesting but somewhat perplexing results of these studies.

RESPONSE FORMAT 169

Acknowledgments

The authors wish to thank the following persons for their valuable assistance in the completion of these studies and this manuscript: Icek Ajzen, University of Massachusetts, Amherst; Terry Shimp, University of South Carolina; Paul Min- iard, Ohio State University; Don Lehmann; and two anonymous reviewers.

Notes

1. Although Liefeld (1988) found some differences among responses for CAQS, personal interviews and self-completion questionnaires, most of the observed differences were attributed to differences in response protocols (i.e., moving the cursors, initial cursor placement) and should not be factors in this study since the CAQ method required no use of cursor or other control keys. Response for the CAQ method simply required the respondent to type the number associated with their response while the paper-and-pencil method required the respondent to circle the number.

2. The primary concern in this study is the response format effects and not the theory of reasoned action, per se. For this reason, we focus only on the attitude-intention relationships and do not explicitly consider subjective norms or the model itself.

3. Thirteen belief statements were included in the questionnaire for donating blood. Since initial analysis of responses indicated that two of the items had item-to-total correlations of less than .20, these two items were eliminated from subsequent analyses.

References

Ajzen, Icek, and Martin Fishbein. (1980). Understanding Attitudes and Predicting Social Behav- ior. Englewood Cliffs, NJ: Prentice-Hall, Inc.

Bagozzi, Richard P. (1982). “A Field Investigation of Causal Relations Among Cognitions, Affect, Intentions, and Behavior,” Journal of Marketing Research 19 (November), 562-584.

Beckwith, Neil E., and U. Victor Kubilius. (1978). “Empirical Evidence of Halo Effects in Store Image Research by Estimating True Locations, ‘* Advances in Consumer Research 5,485-493.

Beckwith, Neil E., and Donald R. Lehmann. (1975). “The Importance of Halo Effects in Multi- Attribute Attitude Models,” Journal of Marketing Research 12 (August), 265-275.

Beckwith, Neil E., Harold H. Kassarjian, and Donald R. Lehmann. (1978). “Halo Effects in Mar- keting Research: Review and Prognosis, ” Advances in Consumer Research 5, 465467.

Chan, Florentius, Joseph Karbowski, Richard A. Monty, and Lawrence C. Perlmuter. (1986). “Performance as a Source of Perceived Control,” Motivation and Emotion 10 (1), 59-70.

Cooper, William H. (1981). “Ubiquitous Halo,” Psychological Bulletin 90 (2), 218-244. Fazio, R. H., T. M. Lenn, and E. A. Effrein. (1984). “Spontaneous Attitude Formation,” Social

Cognition 2, 217-234. Feldman, Jack M., and John G. Lynch, Jr. (1988). “Self-Generated Validity and Other Effects of

Measurement on Belief, Attitude, Intention and Behavior,” Journal of Applied Psychology 73 (3), 4211135.

Laroche, Michel. (1978). “Four Methodological Problems in Multi-Attribute Attitude Models,” Advances in Consumer Research 5, 175-179.

170 PAM SCHOLDER ELLEN AND THOMAS .I. MADDEN

Liefeld, John P. (1988). “Response Effects in Computer Administered Questioning,” Journal of Marketing Research 25 (November), 405-409.

O’Brien, Terry, and Valerie Dugdale. (1978). “Questionnaire Administration by Computer,” JOLIY- nal of the Market Research Society 20(4), 228-237.

Sheppard, Blair H., Jon Hartwick, and Paul R. Warshaw. (1988). “The Theory of Reasoned Action: A Meta-Analysis of Past Research with Recommendations for Modifications and Future Re- search,” Journ& of Consumer Research 15 (December), 325-343.

Tesser, Abraham. (1978). “Self-Generated Attitude Change.” In L. Berkowitz (ed.), Advatzces in Experimental Social Psychology, New York: Academic Press. 289-338.

Thorndike, E. L. (1920). “A Constant Error in Psychological Ratings,” Journal ofApplied Psy- chology 4, 25-29.

Tourangeau, Roger, and Kenneth A. Rasinski. (1988). “Cognitive Processes Underlying Context Effects in Attitude Measurement,” Psychological Bulletin 103(3), 299-314.

Wells, F. L. (1907). “A Statistical Study of Literary Merit,” Archives ofPsychology 1 (7). Wilkie, William L., John M. McCann, and David J. Reibstein. (1973). “Halo Effects in Brand

Belief Measurement: Implications for Attitude Model Development,” Advances in Cowumer Research 1, 280-290.

Wilson, Timothy D., Dana S. Dunn, Dolores Kraft, and Douglas J. Lisle. (1989). “Introspection, Attitude Change, and Attitude-Behavior Consistency: The Disruptive Effects of Explaining Why We Feel the Way We Do.” In L. Berkowitz (ed.), Advances in Experimental Social Psychology. San Diego, CA: Academic Press, 287-343.

the impact of response format on relations among intentions, attitudes, and social norms

Documents