THE VALUE OF EVENTS AND TOURISM:

Expanding the Frontiers of Evaluation and Impact Assessment

Presentation by Donald Getz, PhD. The University of Gothenburg, Sweden,

March, 2014 1

TOPICS • A Retrospective • Forces, Trends and Issues • Some Theory and Definitions

– Evaluation and the worth of events/tourism

• Beyond the Triple Bottom Line – The Event Compass: goals driven,

comprehensive, flexible – A system for taking event and

tourism evaluation forward – Proposal for a new international

Delphi study

2

A Retrospective Gaze The relevant (English-language) literature had its origins in the context of rapid growth in international tourism in the 1960s and 1970s. Early consideration of event impacts came from various disciplinary perspectives, creating a de facto field of Tourism Studies.

• John Lickorish & Andrew Kershaw (1958) wrote The Travel Trade including a discussion of multiplier

effects.

• Historian Daniel Boorstin (1961) first drew attention to the phenomenon of ‘pseudo events’ • Greenwood’s (1972) anthropological study of a Basque festival raised lasting questions about

“commodification”.

• Tourism – Blessing or Blight? Sir George Young (1973) was very influential in stirring up debate about mass tourism impacts.

• Anthropologist Victor Turner’s work on pilgrimage (1978), liminality and communitas (1974) have had tremendous influence on tourism and event studies with regard to social and cultural meanings and impacts.

• Sociologist Dean MacCannell’s book The Tourist: A New Theory of the Leisure Class (1976) is always cited in discussion of authenticity.

3

From the Tourism Journals • Estimating the economic impact of an event through visitor expenditure studies

and the use of income multipliers: Vaughan's (1979) study of the 1976 Edinburgh Festival and Della Bitta et al. (1978) who reported on a Tall Ships event.

-Economic impacts and the role of Hallmark Events in overcoming seasonality: J.R.B. Ritchie and Beliveau (1974) on the Quebec Winter Carnival -J.R.B. Ritchie’s (1984) treatise on the nature of social impacts of Hallmark Events

4

Rapid Acceleration in the 1980s • Event Tourism expanded dramatically as a research topic in the 1980s. A number of extension studies at Texas

A&M focused on events and tourism including the Gunn and Wicks (1982) report on visitors to a festival in Galveston.

• Internationally, the AIEST (1987) conference produced a notable collection of material on the general subject

of mega events.

• The 1985 TTRA Canada Chapter conference was themed ‘International Events: The Real Tourism Impact’ (Travel and Tourism Research Association & Canada Chapter (TTRA), 1986), with the impetus coming from the planned 1986 Vancouver World’s Fair and the 1988 Calgary Winter Olympics.

• A major study of festival visitors and the economic impacts of multiple festivals in Canada’s National Capital Region was conducted in the latter part of this decade (Coopers and Lybrand Consulting Group, 1989), followed by a similar major study in Edinburgh (Scotinform Ltd., 1991). These remain landmarks in terms of their scope and cross-event comparisons.

• Australian scholars have always been prominent in event-tourism research. Prior to the America’s Cup Defence in Perth in 1988, the People and Physical Environment Research Conference, 1987, was held under the theme of the Effects of Hallmark Events on Cities. Soutar and McLeod (1993) later published research on residents’ perceptions of that major event.

• One of the most influential research projects of that period was the comprehensive assessment of impacts from the first Adelaide Grand Prix (Burns, Hatch, & Mules, 1986; Burns & Mules, 1989). At the end of the 1980s, Syme, Shaw, Fenton, and Mueller (1989) published a book entitled The Planning and Evaluation of Hallmark Events.

5

The Three Discourses

EVENT STUDIES

EVENT MANAGEMENT

EVENT TOURISM

an interdisciplinary field that studies all planned events, and meanings attached to events and their experience (encompassing event management and event tourism)

an applied field devoted to understanding and improving tourism through events

an applied, professional field devoted to understanding and improving the management of planned events

EVENTS in CLOSELY-RELATED FIELDS: The study of events within other applied fields, including hospitality, leisure, sports, arts, theatre, cultural studies

6

THE NORDIC SCHOOL

• A long-standing tradition of event-studies scholarship in Scandinavia.

• The Impact of Mega Events: papers of the Talk at the Top Conference, 1997, Mid Sweden University, Östersund, Sweden

• Lena Mossberg (Ed.). (2000). Evaluation of events:

Scandinavian experiences. New York: Cognizant.

7

EVENTS BEYOND 2000: SETTING THE AGENDA

PROCEEDINGS OF CONFERENCE ON EVENT EVALUATION, RESEARCH AND EDUCATION, SYDNEY

Edited by: John Allen, Robert Harris, Leo K Jago and A J Veal

J.J. Goldblatt: “…unless the profession adopts national, or even better international standards for the evaluation of event impacts, it will be difficult for the professionals to be seriously respected and valued by those who control funding and other critical resources needed to support event growth. D. Getz: “What Is an Event Worth? …We need to know what consumers think a given event is worth to them in terms of money and time expended. In other words, what benefits do they seek, how else can they obtain them, and what are they willing to give up to attend any particular event? We could use 'willingness to pay' to answer the question, although many events are nominally 'free' or priced at non-market levels, which complicates the issue.” Larry Dwyer, Robert Mellor, Nina Mistilis, Trevor Mules: A Framework For Evaluating And Forecasting The Impacts Of Special Events “Consistent use of survey instruments which yield meaningful data will enable Fact Sheets to be continually updated, with average values re- calculated. …The limitations of the data relating to ‘intangibles’ are even greater and existing studies have either ignored these impacts or treated them very superficially.

8

PRE-EVENT AND POST-EVENT EVALUATION CRITERIA

RESEARCH Carlsen, J., Getz, D., and Soutar, G., 2001. “Event Evaluation Research”, Event Management, 6 (4): 247-

257. • This paper describes the results of an investigation of the current

use and importance of evaluation criteria by tourism destination authorities (TDAs).

• The study involved a Delphi survey of Australian and international experts in event evaluation from the public and private sector.

• Both pre-event and post-event evaluation criteria were investigated using a Likert scale and conjoint analysis.

• The importance of alternative methodological issues including return on investment (ROI), use of multipliers and the issue of time switching and event-related travel was also investigated.

9

FORCES PROPELLING AND CONSTRAINING EVENT TOURISM

CONSTRAINING Climate change; energy supply; event

costs; political turmoil; recessions; anti-globalization values; poverty and disparity; health concerns; water/food

shortages

PROPELLING Globalization: the economy,

technology, migration; population & economic growth; ease of travel;

competition among destinations and corporations; instrumentalism

LEGITIMATION GROWTH COMPLEXITY

10

TRENDS and ISSUES • Complexity is ever increasing: multiple goals and stakeholders are

involved; A multi-stakeholder, multiple perspective system for evaluation of tourism and events is needed

• Competition for event tourism

• Accountability needs are high; gigantism; competitiveness; the costs and benefits are substantial

• Proliferation of events continues unabated; enormous growth in the sport and leisure realms

• Gigantism and out of control costs; white-elephant legacies; debt • Many questions raised about justifying public intervention; the

role of politics and ideology 11

TRENDS and ISSUES • Critical discourse: academics, media, public: what are

events and tourism worth? To whom? Communities favour imposing a social license to operate

• Greening of events taken for granted – except the biggest, which can never be green and sustainable

• Institutionalization of periodic events: as valued traditions that meet multiple community goals; Hallmark Events to the fore

• Legitimation: events have never been valued more

12

Integrating Event Tourism With Other Policy Domains

EVENTS and PORTFOLIOS

PLACE MARKETING: Co-branding events with

destination

ENVIRONMENT -Heritage and nature

conservation -Urban renewal

-Environmental education through events

TOURISM

Image Maker: Create positive image

of destination

Animator: Of venues, spaces,

resorts, cities

Catalyst: For new infrastructure and increased tourism

capacity

Attraction: -High-yield event tourists

-All year round -Geographic spread

SOCIAL -Foster social integration -Improve quality of life -Capacity building for

community development

CULTURAL -Arts and cultural development -Preserve authentic traditions

13

Justification of ‘Public Good’ • Any government intervention (i.e., direct or indirect

assistance to tourism and events) must be based on claims of creating or sustaining ‘public good’.

• Most economic impact assessments ignore this question altogether or limit their consideration to a narrow estimate of economic ‘benefits’.

• Its up to politicians and an informed public to decide if public good justifies the intervention, relative to other possible uses of resources.

14

JUSTIFYING INTERVENTION IN EVENTS AND TOURISM The Imputed Benefits that Lead to Goals

• ECONOMIC: Tourism, place marketing, urban development and renewal, regional development; new infrastructure to be shared with residents or as a catalyst for growth/modernisation

• Market Failure explains why many governments feel the need to intervene while in some jurisdictions there is an ideological basis for providing Social Tourism and events as entertainment, leisure and culture.

• SOCIAL AND CULTURAL: identity; social integration; health benefits through participation in arts, leisure, sport; preservation of traditions; psychic benefits and personal well being; capacity building for community development

• ENVIRONMENTAL: contributing to education and conservation; the greening of events

15

POPULATIONS OF EVENTS • Organization Ecology can be applied to the study of

festivals and events

• Implications for evaluation and impact assessment: – Environmental forces can affect the health and viability of

populations and individual events – they are not un full control of their destiny

– Inward-looking evaluation often fails to consider major forces like competition, economic cycles, consumer trends, or political shifts

– The impacts of events and populations of events are cumulative and long-lasting; often unpredictable

16

PORTFOLIO MANAGEMENT

Agencies that produce, fund, assist or promote events are building portfolios Implications for Evaluation: • All ‘managed’ events are assets for the destination and the

community; what is the asset value of an event? • Multiple, overlapping portfolios are the new norm • Long-term, cumulative impacts of portfolios have not been

examined • Lack of theory regarding evaluation of portfolios

17

MANAGING AN EVENT-TOURISM PORTFOLIO

HALLMARK EVENTS Permanent institutions;

co-branded with destination

ONE-TIME MEGA-EVENTS

WON THROUGH BIDDING

OTHER ANCHORED

EVENTS THAT CAN BE GROWN

ONE-TIME EVENTS WITH POTENTIAL TO ANCHOR,

RETURN, GROW

ASSET VALUE

COSTS; RISKS

18

Evaluation and Valuation • There are two widely recognized meanings of evaluation

(source: www.socialresearchmethods.net):

• 1: Evaluation is the systematic assessment of the worth or

merit of some object

• 2: Evaluation is the systematic acquisition and assessment of information to provide useful feedback about some object

• Whereas 'evaluation' is often used in the context of program or policy evaluation, with the specific aims of proving cause and effect (i.e., did the program achieve its aims?) or of demonstrating how to make improvements in terms of effectiveness and efficiency, the term 'valuation' can have a more specific meaning.

19

Valuation • While determining economic value most certainly requires a monetary measure, the question of 'worth' is much more subjective. • Value or worth can be expressed as opinion rooted in

ideology, or perception of costs and benefits.

• Whether or not an event or tourism has 'merit' will inevitably reflect the judgment of residents, politicians and other stakeholders, all of whom can be ill informed or biased.

20

WHY EVALUATE? -to justify the event; legitimacy building

-to assign value/worth as assets -to aid in planning/design/marketing; solve problems; clarify logic and goals

-to improve management systems -to improve competitiveness; stay on

track; be successful -to determine outcomes/impacts

WHAT TO EVALUATE? -policies and strategies

-environmental conditions/trends/competitors -organizational structure/stakeholders

-management practices -event operations

-impact causes and effects -event portfolio worth and population health

MEASURES -efficiency and effectiveness

-quality; competence -economic, social, cultural, environmental

impact indicators -ROI

-opportunity costs; externalities

METHODS -observation & self-reporting

-systems approach -visitor and stakeholder input

-experimental research designs -field research designs

-action research

USES -constant monitoring & feedback

-input to project and strategic planning -community & stakeholder engagement

-inputs to certification

EVALUATION PARADIGMS: -positivism -interpretive -critical/emancipatory

EVALUATION CONTEXTS: - formal versus informal - performance and goal driven versus goal-free/systems approach - accountability required versus independence - internal versus external and multi-stakeholder - collaborative, participatory versus independent - project-based versus long-term sustainability - single event versus portfolio - one-time event versus periodic 21

Evaluation Paradigms • According to Rossi et al (2004) the positivist paradigm

dominates contemporary evaluation research, notably through the prevailing emphasis on measuring economic impacts, customer satisfaction, and other quantifiable outcomes.

• According to Rossi et al (2004) it includes needs assessment, assessment of program theory and process, and efficiency.

• In contrast, a set of interpretive approaches has been described by Potter (2006), and these have in common an attempt to work with stakeholders to understand their expectations, experiences and meanings before making judgment about value or worth.

22

Interpretive Evaluation Wood (2009: 183) put forward a framework for the evaluation of festival impacts that can be considered interpretive in design. It was "intended to incorporate a wider view of evaluation than has been used so far in event evaluation. The framework suggests that the views of all stakeholders are considered, and that evaluation takes place: • within specific and unique contexts (and therefore cannot be standardized)

• a wide range of methods and sources of information is required to give a

full and balanced view

• outcomes or objectives although important are not the only aspects to evaluated

• a cost– benefit analysis or economic evaluation is not suitable for this type of festival where many of the effects are intangible, social, and long term.

23

The Measurement Problem • There is always a measurement problem in science, starting with the rubrics that

something does not exist unless we can measure it….…and everything is affected by the measurement process!

• In other words, unless we have measures of the cultural impacts of events we cannot say if they exist or not.

• Second, by using a particular measure, such as 'growth in arts participation' as a measure of festival success, evaluators can either deliberately or inadvertently shape future design and consequential attendee experiences.

• The triple-bottom-line approach suffers from the problem of incommensurability, namely each of the three lines of impacts require different measures; economists like to use monetary measures across all three:

Andersson, T., & Lundberg, E. (2013). Commensurability and sustainability: Triple impact assessment of a tourism event. Tourism Management, 27, 1-11.

24

ASSESSING AN EVENT’S “FOOTPRINT”

From E. Fredline et al 2005.

SOCIAL IMPACT SCALE

ENVIRONMENTAL IMPACT SCALE

ECONOMIC IMPACT SCALE

In this example the event is assessed as having low economic, medium social, and high environmental impact . Issues: -what does each scale measure? (indicators) -will each scale be given the same “weight”?

low

high

medium

0

8

5

3

25

Theory-Based Evaluation • For events and tourism, we have predictive theory

that shapes evaluation: i.e., the process by which new money (i.e., incremental tourist expenditure and other investments) generates value for an area.

• This ‘theory’ is supported by much research on tourist motivation, consumption and expenditure behaviour, and the multiplier effect)

• Using this theory enables researchers to forecast economic impacts and measure them during or after an event.

26

Theory-Based Evaluation

• Reasonably sound theory exists to make forecasts about the following: – How tourism development generates jobs, attracts

in-migration, creates new leisure and entertainment options, and consequently changes the social environment

• What about the links between tourism, events, and: – Social and cultural integration; capital; health?

27

Evaluation Lacking Theory • Other than economic impacts there is little or nothing

in the way of theory that enables evaluators to predict what the outcomes of tourism or events will be with any certainty.

• In the absence of theory we see the application of ideology and development or marketing dogma, wishful thinking, and vague policies or strategies.

• Many evaluations are also plagued by deliberate weaknesses: no control groups, no pre and post measures, no accountability.

28

The Event Compass™ Story

29

Event Compass - www.miun.se

Internal – Process Dimensions

External - Impact Dimensions 30

APPLICATIONS OF THE EVENT COMPASS

• Evaluate management: – Set goals and measure goal achievement; then revise goals along each

dimension; a tool in strategic planning and decision making

• Perform impact assessments (events, portfolios) – All 8 dimensions come with KPIs, measures and methods in a toolkit

• Perform feasibility and risk assessments when making event bids or other projects.

• Certify events (for funding or marketing purposes) – Specify minimum and desirable achievements along each dimension (similar to

Green Leaf)

31

KEY PERFORMANCE INDICATORS AND METHODS

32

ORGANISATION

PLANNING

DESIGN

ENVIRONMENTAL OUTOMES

ECONOMIC OUTCOMES

0

20

40

60

80

100

MARKETING

RISK

SOCIO-CULTURAL OUTCOMES

EVENT COMPASS EVALUATION SYSTEM

33

Selected Research and Theory Building Related to the Valuation of Tourism and Events

• Social Impact Assessment • Resident perceptions and attitudes • Stakeholder perceptions and positions • Social leveraging • Use and Non-Use valuation • Cumulative impact assessment • Portfolios and Populations

34

Sunshine Coast, Australia • An online survey of residents asking questions about their

attendance at events (including willingness to pay / use values) and their attitudes towards events, venues and strategies (including willingness to pay / non-use valuation).

• This research extended previous work on single events with the goal of applying use/non-use valuation to whole populations of events in one city.

Tommy D. Andersson , John Armbrecht & Erik Lundberg (2012) Estimating Use and Non-use Values of a Music Festival. Scandinavian Journal of Hospitality and Tourism, 12:3, 215-231.

Tommy D. Andersson & Erik Lundberg (2013). Commensurability and sustainability: Triple impact assessments of a tourism event. Tourism Management, 37, 99-109.

35

Sunshine Coast Findings • Residents of the Sunshine Coast greatly

value their planned events, with substantial direct use value accruing from attendance, active participation (e.g., as athletes and artists), and volunteering.

• Residents gave strong recognition to the economic, personal and social values of planned events, regardless of whether they used them or not.

• Non-use values recognized by residents include 'option value' (i.e., having more to chose from); 'bequest value' (i.e., valued traditions; creating a legacy for future generations; more for youth to do), and existence value (i.e., events are good for the community in multiple ways).

36

Sunshine Coast Findings • Residents largely believe that planned events

are not causing significant social or environmental problems.

• From a policy perspective, planned events represent a major field in which both economic gain and general public good are being created, with even greater potential to grow.

• Residents especially support festivals and cultural celebrations, and a range of events and event venues are also supported.

• There is only limited support among residents to accept tax increases to pay for new events or venues; this could be controversial, and a full justification for new capital expenditures and ongoing support should always be provided to residents.

37

CONCLUSIONS

• Although many of the issues and topics remain the same, the evaluation challenge has never been more complex; demand for accountability and sustainability is increasing.

• Impact assessments are but one aspect of evaluation; they do not in themselves infer worth or value.

• The Triple-Bottom-Line approach is merely a starting point for comprehensive thinking; incommensurability along three dimensions remains a serious issue.

38

CONCLUSIONS

• The Event Compass represents a way forward for evaluations, impact assessments and certification.

• Theory-based evaluation in tourism and event management is weak; proving cause and effect is always going to difficult in the qualitative dimensions, hence the value of a goal-attainment system.

39

HOW SHOULD ‘VALUE or WORTH’ BE DETERMINED?

Assuming an open, democratic context

• Only a multi-stakeholder approach will work in the long run • It should be a transparent political process • Will have to consider populations, portfolios and cumulative, long-term impacts • It should be goal driven, justified by integrated policies (not solely tourism or economic development) …. • in which the roles of events and tourism are fully articulated • Aided by full documentation of risks, costs and benefits It is in this context that the Compass has its greatest potential as a tool in consensus building, goal setting, and comprehensive evaluation of events , portfolios and populations - as a permanent process.

40

END

41