Convince them its real or tell them its hypothetical?

The Effect of Alternative Explanations for Implausible

Alternatives in Choice Experiments

Mark Morrison and Olvar Bergland

The problem of hypothetical bias

• In CVM and CM there is an emphasis on ensuring that the scenarios presented to respondents are as real and consequential as possible

• Otherwise can lead to hypothetical bias and unreliable results

Mitchell and Carson (1989)

• Suggested several factors that affect the realism of a hypothetical scenario, including respondents:– being familiar with the key scenario elements (eg

the good being valued, and the payment vehicle)– being able to understand the scenario, and – finding the scenario plausible.

Arrow et al (1993)

• “If CV surveys are to elicit useful information about willingness to pay, respondents must understand exactly what it is they are being asked to value…and must accept the scenario in formulating their responses”.

Experimental evidence

• Cummings and Taylor (1998) demonstrated that it is only when respondents “place somewhat high probabilities on the realness of the survey”, that hypothetical and real estimates converge.

Implausible/Unlikely Alternatives in Choice Experiments

• Arise from the use of orthogonal designs• Hard to always understand what respondents

will consider implausible• Can be reduced by the use of correlated/

nested designs

Options for explaining implausible/unlikely alternatives

• Many studies have sought to explain why implausible/unlikely alternatives are possible despite focus group results indicate that many doubt the veracity of these comments (eg Blamey et al 1997; Morrison, Bennett and Blamey, 1997, Morrison and Bennett 2004).

• Others have admitted that they are hypothetical and explained why they are being asked (eg Mallawaarachchi et al 1999, Morrison 2000).

Examples

• The outcomes in each of the options have been specifically defined so that you have a broad range of choices. Within this range, some options may seem strange according to your experience, but bear in mind that there are many different ways of managing water. For example, wetland vegetation requires regular floods of various sizes to remain healthy. In contrast, waterbird breeding requires flooding of sufficient size at a suitable time of year (Morrison, Bennett and Blamey 1999).

Examples

Some of the options may also seem a little odd – for example an option that involves less area and more native species. However, such a possibility could occur if the smaller area is all high quality habitat, and some of the larger area is of lower quality (Lockwood and Carberry 1998).

Examples

• To keep matters simple, we do not describe how each management option would work. For example, we do not specify how trees would be protected from clearing, and which areas of the Desert Uplands would be protected. Rather, we simply describe what effect it would have on jobs, the natural environment etc. Indeed, there are a lot of ways that the issue of tree clearing can be managed. When answering the questions below, please keep in mind that some options which seem a little odd may in fact be quite possible (Blamey, Rolfe, Bennett and Morrison 2000).

Examples

• We would now like to find out your views about some options for the use of land in the Herbert River District. To do this we have prepared eight sets of hypothetical options that produce a broad range of possible land use outcomes. In each question, Option A describes current practice and Options B and C describe the outcomes associated with two possible incentive schemes. By indicating the option you prefer in each of these sets, we will be able to identify the outcomes you think are most important, and the land use options that best suit people like yourself (Mallawaarachchi et al 1999).

Examples

• We are interested to find out your views on some possible management options for the Macquarie Marshes. To do this, we have prepared eight sets of hypothetical options. From each set of options we would like you to choose the option you prefer…These options have been designed to cover a broad range of possible wetland management options. By indicating the option you prefer in each of these sets, we will be able to identify the outcomes that you think are most important, and the options that best suit people like yourself (Morrison 2000).

What are the likely effects of these alternative approaches?

• If the admissions made are congruent with respondents’ beliefs about the information provided, respondents will be more likely to take the survey seriously, thereby reducing bias and variance

• May increase strategic behaviour

Kelman (1967)

Deception may cause the subject to dismiss the stated purpose of the study and search for other interpretations. May cause the subject to receive contradictory messages– if a contradiction between the researcher’s

statements about the purposes of the study and the information the subject receives from the experimental conditions

Bailey (1978)

• May cause different subjects to have their own definition of the meaning of the questionnaire and thus adhere to different response strategies.

• ‘This is somewhat analogous to survey respondents having various understandings of an ambiguous question, so that in a real sense different respondents are answering different questions while all are ostensibly answering the same question” (pp.390-391).

Case Study

• Land use change in Sunshine Coast area of Queensland

• Sunshine Coast has a very high rate of urban development– Threatening the viability of the Sugar Cane Industry– Affecting areas of natural vegetation– Three competing priorities

• Except for natural vegetation, no strong a priori view about whether willingness to pay would be positive or negative

Survey logistics

• Questionnaire was sent to 825 respondents on the Sunshine Coast

• Mail survey was used – initial questionnaire plus two reminders

• 40.7% response rate was achieved (312 valid responses)

Sample characteristics

Mean

Age 53.5 years

Sex 48% female

Education* 4.5

Income $42,148

* 1-never went to school, 6-tertiary degree

Questionnaire description

• 20 page booklet • Accompanying the questionnaire was a brochure

that provided detailed information about the three main land uses in the Sunshine Coast region.

• After several introductory questions respondents were asked four contingent ratings questions.

• The first alternative that each person rated was the status quo, then they rated three alternatives that were different to the status quo.

Questionnaire description

• Ratings scale in the form of a Likert scale. • Labels and small number of points on the

ratings scale likely to reduce differences in the way respondents use ratings scales

• Use of support/oppose labels makes use of the ratings scale much more like choice

Questionnaire description

To start off with, we would like you to evaluate some land use options in terms of whether you support or oppose the region moving in this direction. The first option we would like you to consider is the continuation of existing policies. Question 4 Urban Area in 2010 Area of sugar cane in

2010 Area of rare or unique vegetation in 2010

Change in land rates

19,000 hectares

5000 hectares

15,000 hectares

No change

What do you think of this option? tick one box only I strongly support this option I support this option I neither support nor oppose this option I oppose this option I strongly oppose this option

• First respondents answered four ratings questions

Questionnaire description• Next respondents answered six trinary choice questions

Question 8: Suppose the following three options were the only ones available, which ONE would you choose? Option A: Continue existing policies Urban Area in 2010 Area of sugar cane in

2010 Area of rare or unique vegetation in 2010

Change in land rates

19,000 hectares

5000 hectares

15,000 hectares

No change

Option B: New option Urban Area in 2010 Area of sugar cane in

2010 Area of rare or unique vegetation in 2010

Change in land rates

22,000 hectares

5000 hectares

26,000 hectares

$200 increase

Option C: New option Urban Area in 2010 Area of sugar cane in

2010 Area of rare or unique vegetation in 2010

Change in land rates

12,000 hectares

5000 hectares

17,000 hectares

$50 increase

I would choose tick one box only Option A: Continue existing policies Option B Option C Not sure

Two Treatments (1)We would now like to find out your views about some options for the use of land on the Sunshine Coast. To do this, we have prepared a number of hypothetical options for you to evaluate. By evaluating these options, we will be able to identify the outcomes you think are most important, and the options that best suit people like yourself.

You will need to read the enclosed brochure before evaluating these options. The brochure describes the features of the options we would like you to consider in this section.

The options have been specifically designed so that you have a broad range of options to consider.

Two Treatments (2)We would now like to find out your views about some options for the use of land on the Sunshine Coast. To do this, we have prepared a number of options for you to consider.

You will need to read the enclosed brochure before evaluating these options. The brochure describes the features of the options we would like you to consider in this section.

The options have been specifically designed so that you have a broad range of options to consider. Some options may seem a little odd, but bear in mind that there are a number of ways of determining land uses.

Results

Hypothetical

version

Standard

version

2 (p-value)

I objected to some aspects of the

questionnaire (5-strongly agree,

1-strongly disagree)

2.61 2.79 11.098 (p=0.025)

I found answering questions 7 to

13 meaningful (5-strongly

agree, 1-strongly disagree)

3.57 3.47 11.433 (p=0.022)

Comparison of ratings models (full ratings model, ordered logit)

Treatment 1(non-hypo)

Treatment 2(hypo)

Urban -0.05(-2.63)***

-0.08(-3.90)***

Sugar 0.53e-2(0.43)

0.31e-1(2.32)**

Vegetation 0.46e-1(2.17)**

0.49e-1(2.28)**

Rates -0.45e-3(-0.45)

-0.47e-2(-4.62)**

rho2 0.011 0.046

Note: t-stats in brackets, *** sig at 1%, ** sig at 5%, LR Test: X2(8)=18.17, p=0.02

Comparison of ratings models (ratings recoded to choices)

Treatment 1(non-hypo)

Treatment 2(hypo)

Urban -0.80e-2(-0.38)

-0.69e-1(-3.12)***

Sugar 0.39e-2(0.31)

0.12e-1(0.93)

Vegetation 0.32e-1(1.61)

0.77e-1(4.09)***

Rates -0.19e-2(-1.89)*

-0.69e-2(-6.25)***

rho2 0.004 0.049

Note: t-stats in brackets, *** sig at 1%, ** sig at 5% %, LR Test: X2(5)=15.62, p=0.008

Comparison of choice modelsTreatment 1(non-hypo)

Treatment 2(hypo)

ASC -0.79e-1(-0.69)

-0.97e-1(-0.92)

Urban -0.61e-1(-5.90)***

-0.60e-1(-5.70)***

Sugar 0.47e-2(0.81)

0.14e-2(0.37)

Vegetation 0.77e-1(6.79)***

0.74e-1(6.87)***

Rates -0.60e-2(-11.44)***

-0.47e-2(-8.79)***

rho2 0.066 0.048Note: t-stats in brackets, *** sig at 1%, ** sig at 5% %, LR Test: X2(5)=3.56, p=0.61

LR Tests for Ratings vs Choice Models

X2 Value P value

Ratings Models vs Choice Models

Non-hypo 24.89 0.0001

Hypo 3.51 0.4760

Ratings (recoded as choices) vs Choice Models

Non-hypo 17.30 0.0017

Hypo 2.29 0.6828

Implicit Prices

Implicit Prices

Final Observations• Admitting alternatives were hypothetical and explaining why

they were asked was more meaningful and less objectionable to respondents

• Lead to (1) robust ratings results, and (2) convergent validity between ratings and choice results

• Little evidence of strategic behaviour• Choice results were equivalent regardless of treatment

– Are choice models less affected by the explanation given for implausible/unusual alternatives?

– Effect of choice tasks being asked after the ratings questions?

• Equivalence of implicit prices for ratings and choice tasks in Treatment 2 promising for ratings