44

S t r at e g y

Anna Whicher, Research Officer, International Institute of Design Policy & Support, Design Wales, Cardiff

Gisele Raulik-Murphy, Senior Researcher, International Institute of Design Policy & Support, Design Wales, Cardiff

Gavin Cawood, Operations Director, International Institute of Design Policy & Support, Design Wales, Cardiff

The thesis here is that design’s value should be a rich equation, measured within companies, across industries, as the result of design policies, and as part of the economy and society as a whole.

45© 2011 The Design Management Institute

In Europe, design has received more and more attention at the policy level. In 2009, a European Com-mission survey asked about serious barriers to the better use of design in Europe. The most significant obstacle was considered to be “lack of under-standing of design among policy-makers.” The second was “lack of knowledge and tools to evaluate the rate of return on design investment.”2 Following the consultation, in Octo-ber 2010, design was highlighted as a

2. European Commission, “Results of the Public Consultation on Design as a Driver of User-Centered Innovation,” Brussels, 2009.

practical for providing concrete input for informed and strategic policy-making. This is particularly significant at a time when design is rising up the policy agenda. Due to a myriad of converging factors, design policies are emerging and maturing across the globe. Not least among these factors is awareness of successful cases in which design has been integrated into a government strategy for economic growth. Asian and Scandinavian examples are among the most promi-nent, as was demonstrated in the most recent issue of this journal.

An increasing body of knowledge asserts the positive contribution of design to economic growth. In recent years, researchers and practitioners have strived to evaluate the impact of design at micro and macro levels in comparative studies around the world.1 Despite encouraging results, some of these methods remain im-

1. Designium, “Global Design Watch 2010,” SEE bulletin issue 5 (2011), University of Wales Institute, Cardiff, pp. 3-5; APCI, “Economie du Design,” Paris, 2010; Design Council, “Design Industry Insights 2010,” London, 2010; H. Hollander and A. Van Cruysen, “Design, Creativity, and Innovation: A Scoreboard Approach,” Pro Inno Europe, 2009; J. Moultrie, “Developing an International Design Scoreboard,” SEE bulletin issue 1 (2009), University of Wales Institute, Cardiff, pp. 3-6; KIDP, “National Design Competitiveness Report 2008,” Seoul, 2008.

Evaluating Design: Understanding the Return on Investmentby Anna Whicher, Gisele Raulik-Murphy, and Gavin Cawood

Design Management Metrics: assessing Quality and Outcomes

46

Differentiating these dimensions is vital for informing policy-makers about the different aspects of design practice. We note that evidence on the impact of design investment is needed at various levels—from investments made by single companies for their individual profit to the impact resulting from a policy that promotes design nationally. In proposing this framework (Figure 1), we intend to help researchers to un-derstand these various dimensions and encourage them to conduct studies that will form a richer body of knowledge on the value of investments in design. Each dimension is illustrated by a case study representing current practice.

able to stand up to rigorous scrutiny.This article seeks to provide an

overview of current practice in design evaluation and proposes a number of dimensions that must be taken into consideration when evaluating design at the micro and macro level in both private and public sectors:

• Return on investment in individual companies

• Return on investment in national industry

• Return on investment in design programs and policies

• Return on investment in economy and society

priority in the Europe 2020 Flagship Initiative Innovation Union.

As part of the Europe 2020 strategy, the European Commission has set three interconnected goals for future growth—it should be smart, sustainable, and inclusive—and stated that “design can contribute to all three factors.”3 With design now firmly on the European political agenda, policy-makers across Europe are looking to understand its role in innovation and its return on investment. If design is fully to justify its emerging profile at a policy level (particularly in Eu-rope), researchers must answer a few fundamental questions: What are the challenges associated with evaluating design? How does design enhance a company’s competitiveness? Is in-dustry taking advantage of design re-sources? Has government investment in design programs and policies paid off? To what degree is design contrib-uting to national development?

As design climbs the policy agenda, the importance of addressing the evaluation of the return on design is more relevant than ever. Evaluation is a vital part of the evidence to sup-port decision-making, and in the con-text of government cuts—a searing current issue in Europe—needs to be

3. Buescher, R. “Design in the Europe 2020 Agenda,” speech at the Design and Learning conference, Brussels, November 25-26, 2010.

21

43

PRIVATE SECTOR

MA

CR

O LEV

EL

PUBLIC SECTOR

MIC

RO

LEVEL

Levels of Design Evaluation

national industry

individual companies

national economy / society

individual programmes

& policies

Figure 1. Framework representing dimensions in evaluating design. Source: G. Raulik-Murphy, “Evaluat-ing Design—Workshop Introduction,” from SEE workshop “Evaluating Design and Innovation Policies,” Florence, Italy, May 10-11, 2010.

evaluating Design: Understanding the return on I nvestment

47

we can identify that design contrib-utes in two ways—as a sector by itself and as a strategic discipline that contributes to the growth of other sectors in the manufacturing and service industry. For governments to appreciate the significance of design both as a sector and as a strategic

discipline, decision-makers need to have data on both aspects.

Design adds value to individual businesses, and therefore it increases the value of national industry. In order to measure this impact, it is necessary to accumulate data about the

contribution to individual companies (dimension 1) and how it can be multiplied to represent the impact on the industry as a whole. As a sepa-rate sector, the analysis of the design contribution demands data on the size of the design sector itself, its em-ployment distribution, and financial contribution in order to promote an appreciation of the scale of the design industry and its contribution to com-petitiveness. Examples of surveys on the composition of the design sector have been conducted by a number of national design organizations. A selection would include the Seoul Design Center’s “Asia Design Sur-vey 2009,” the UK Design Council’s

16-to-44-year-old male audience. A unique rebranding exercise, which created a gentleman’s club-style ambi-ance, resulted in remarkable growth. Dave leapt from the 29th biggest channel to the 10th and became the largest among the target segment, attracting eight million new view-

ers. A tight investment of £100,000 was transformed into a profit of £4.5 million in the first six months alone, and the channel’s incremental growth contributed a staggering £25 million in ad-sales revenue for 2008. Dave is now rapidly gaining market share at the expense of larger, more-estab-lished channels. This evidence proves the value of the design intervention for this business, proving the impact of design on individual companies is the first step in evaluation.

evaluating the return on design in national industry

When analyzing the ROI the private design sector offers at a macro level,

evaluating the return on design in companies

There has been a drive over the past decade to investigate the return on investment (ROI) of design in indi-vidual companies to prove its com-mercial value, not only to potential clients but also to decision-makers in government. This is probably the dimen-sion that has been best explored under current practice, mainly in the form of case studies published by individual design agencies and national design bodies. The European Com-mission recognizes that “the results are compelling: companies that invest in design tend to be more innovative, more profitable and grow faster than those who do not.”4 Nevertheless, measuring design in statistical terms remains problematic, since evaluation is costly and design’s contribution cannot easily be extracted from the broader commercial context.

As an example of this type of evaluation, we present Dave, winner of the 2008 DBA’s Design Effective-ness Award. In a complex market, a small UK digital TV channel wanted to increase its share of the lucrative

4. European Commission, op. cit., p. 2.

There has been a drive over the past decade to investigate the return on

investment (ROI) of design in individual companies to prove its commercial value,

not only to potential clients but also to decision-makers in government.

Design Management Metrics: assessing Quality and Outcomes

48

2010 “Design Industry Insights,” and Economie du Design, from l’Agence Pour la Promotion de la Création Industrielle.

To highlight a study on measur-ing the impact that design has in the industry as a whole, we have selected a pioneering study conducted be-tween 2003 and 2006 by the Danish Design Centre in association with the National Agency for Enterprise. “Economic Effects of Design” set out to measure the level of design activ-ity in Danish businesses, applying a DDC methodology called the Design Ladder as a way of assessing the eco-nomic benefits of design in Denmark. The study focused on:• Total investment in design• Gross revenue performance and

the development in employment and export share of turnover among the companies

• Difference in gross revenue, em-ployment, and exports for compa-nies that adopt a comprehensive approach to design compared with those who do not use design

The study concluded that Danish companies invested an annual total of approximately DKK 7 billion in design. Overall, companies that in-vested in design showed an additional growth in gross revenue of 250% compared with companies that did

not. Linking performance data with investment in design thus revealed a correlation between design purchase and economic growth. Using the sur-vey data, companies were categorized into four stages of design maturity, depending on their approach to design investment (Figure 2). The higher a company was ranked on the Design Ladder, the greater strategic importance it attributed to design.

The Design Ladder is proving to be a successful tool for evaluat-ing design. However, it is important to highlight that a key issue for a

successful measurement process is its systematic evaluation. Only the collection of data in consecutive periods provides comparative data and, therefore, meaningful results. By assessing how many companies move up a rung on the Design Ladder once design promotion and policies have been implemented, the Danish government was able to make a tangible assessment of the role of design in industry. Not long afterward, it implemented a policy to further support the national indus-try’s use of design.

Figure 2. The Danish Design Ladder.

evaluating Design: Understanding the return on I nvestment

49

evaluating the return on design programs and policies

Here we analyze investments made by the public sector at the micro level—that is, in individual programs and policies for design support and promotion. This is an increasing concern, particularly in Europe, where ROI of public funds is a pertinent issue in light of government cuts. Design programs work to introduce design to individual companies or to encourage industry and the government itself to make better use of design re-sources. Typically, these programs are government-funded, so in this case evaluating their impact is a matter of accountability, as well as of improving the process of delivery.

During the development of this research, the authors were able to work with the group of partners involved in the SEE project—a network of 11 design organizations in Europe working to integrate design into innovation policies. Sampling these 11 design programs, the project managers completed a self-assessment questionnaire to ascertain how effec-tively their design support programs were evaluated. Questions included: What were your program’s targets? How frequently was it evaluated?

How were the delivery and impact of the program measured? How was data collected? What were the con-sequences of evaluation? Using the results of this exercise, we were able to identify some shortfalls that could be improved upon.

Ideally, evaluation needs to take

place at regular intervals to be effec-tive. We observed that on average, the partners’ programs ran from three to five years; however, in five of the elev-en cases, evaluation was conducted only at the end of the program, and only four programs were evaluated annually. The most effective evalua-tion models established benchmark points at the outset to demonstrate the impact of design intervention. We also discovered that significantly more emphasis was placed on the delivery of the program than on its impact. Typically, it was activities that were counted, such as how many seminars/exhibitions had been organized, how many publications produced, how many SMEs were offered advice. (The repercussions of SME involvement in

these programs were not sufficiently captured.)

Although in all cases, the pro-gram of activities was comprehensive, few had set their performance goals in objective, quantifiable, and measur-able terms. Without clear targets, the appropriate data cannot be collected

to assess whether the pro-gram is achieving its goals. For example, many of the objectives were intangible, such as improve coopera-tion between business and academia, enhance the competitiveness of SMEs,

raise awareness of design. Also, many of the positive results, which were not included in the initial goals, were not taken into consideration in the evalu-ation process.

Because some programs are repeated in periodic cycles, it is im-portant that evaluation is taken into consideration at the end of each cycle to ensure improvement. Therefore, the real evaluation challenge is one of emphasizing learning and adapta-tion rather than of merely informing a decision as to whether to continue the program.

In the context of government policies and programs, evaluation needs to be systematic, examin-ing both operations (delivery) and outcomes (effects), comparing results

Here we analyze investments made by the public sector at the micro level—that is, in individual programs and policies

for design support and promotion.

Design Management Metrics: assessing Quality and Outcomes

50

with initial and emerging goals, and using evaluation for ongoing im-provement. Who undertakes this evaluation is open for discussion, but in the short term it will probably be up to those delivering design-related programs to find and undertake appropriate evaluation that will find acceptance with policy-makers over the long term.

evaluating the return on design in economy and society

What is design’s contribution com-pared with that of other sectors and disciplines? As James Moultrie has pointed out, “Whilst there is some evidence to demonstrate the value of design to the firm, there are very few studies that have successfully demonstrated the value of design at a regional or national level.”5 Design’s impact can be multi-dimensional—economic, environmental, social. Cur-rently, this broader dimension of our understanding of design is underde-veloped, since linking national design capabilities with economic perfor-mance entails inherent causality que-ries, particularly due to the scarcity of reliable data. Design is a dynamic tool for the innovation process; while innovation is well measured in many 5. Moultrie, J. (2009) “Developing an International Design Scoreboard,” SEE bulletin issue 1, University of Wales Institute, Cardiff, pp. 3-6.

scoreboards, such as the Community Innovation Survey, design is not simi-larly captured.6 The challenge remains one of how to include questions on design in such surveys. This would allow for a comparative analysis across Europe and beyond. Individually, countries can conduct an assess-ment of design’s contribution to the economy in relation to other sectors.

In Wales, the Creative and Cultural Industries Economic & Demographic Footprint research, developed by the Creative & Cultural

6. As mentioned in Hollander (op. cit.) and Moultrie (ibid.).

Skills council in 2008, is based on data collected from sources including the Annual Population Survey, the Inter-Departmental Business Regis-ter, and the Annual Business Inquiry. It defines the creative and cultural in-dustries in terms of advertising, craft, cultural heritage, design, literature, music, performing arts, and visual arts. In Wales, 24,060 people are employed in the creative industries, and design accounts for the great-est proportion of Wales’ creative and cultural industries (22 percent). The Welsh creative industries contribute £465 million GVA annually to the

Literary Arts 6%

Advertising1%

Cr 15%

Cultural heritage

>1%

Design36%

Music15%

Visual Arts8%

PerformingArts18%

Figure 3. Creative industry productivity levels in Wales: gross value added.

evaluating Design: Understanding the return on I nvestment

51

skepticism. However, this should not halt the progress of policy discussion on the value of design for companies. Indeed, the EU recognizes that the “findings of micro-economic research on design are conclusive—the use of design has a positive impact on the performance of a company, measured in terms of, for example, profitability, share price, employment, or exports.”7 There is a need for pre- and post-measurement across a combination of soft and hard indicators to obtain more evidence of the efficacy of de-sign use within individual companies.

By adding value to individual companies, design adds value to national industry. However, with-out data on the composition of the design sector in terms of employ-7. European Commission, op. cit., p. 25.

• Unclear criteria for success in projects, programs, or policies when objectives are not well defined at the outset

• Difficulties in isolating design’s contribution and impact from the broader context (particularly distinguishing it from traditional measures for innovation)

• Costly evaluation process• Political sensitivity to the results of

the evaluation• Failure to use the evaluation

process as a management tool to improve delivery

Although the case for design’s contribution to business performance is strong, the extent of design’s role as separate from the interaction of other disciplines remains subject to

UK economy, of which 36 percent comes from design (Figures 3 and 4). The value of this research is that it contextualizes the role and contribu-tion of design compared with other creative industries.

Conclusion

As demonstrated in the proposed framework, examining the rate of return on design investment is multi-dimensional and poses many chal-lenges. Among the challenges associ-ated with evaluating design to feed into policy are:• Lack of common definitions and

parameters for the design discipline• Absence of commonly available

measures, indicators, and statistics that can be compared internationally

Cultural Heritage

Literary Arts

AdvertisingDesign

22%

17%14% 13%

11% 11% 10%

2%

Music Cr PerformingArts

VisualArts

Employment in the Creative Industries in Wales

Figure 4. Employment in the creative industries in Wales.

Design Management Metrics: assessing Quality and Outcomes

52

why and when evaluation is needed are paramount in order to raise the awareness and the importance of design on the policy agenda, as well as improve the quality of the strategies

for design practice and promotion. This article identifies the need for an evaluation process that embraces all four aspects of the framework—public and private sectors at both micro and macro levels.

acknowledgements

This article is drawn from SEE Policy Booklet 3: “Evaluating Design”, a publication by the SEE project, which is co-financed by ERDF through the INTERREG IVC program (www.seeproject.org).

The authors would like to thank and acknowledge the contribution made by all the SEE partners who participated at the SEE workshop when this content was initially dis-cussed: Design Flanders, Danish De-sign Centre, Estonian Design Centre, Aalto University, ARDI Rhone-Alps Design Centre, Centre for Design Innovation, Consorzio Casa Toscana, Cieszyn Castle, BIO/Museum of Ar-chitecture and Design, and Barcelona Design Centre. n

Reprint #11222RAU44

steer the operation of the program and may not reflect the intended objectives.

Innovation is well captured in international surveys and scoreboards,

which have enabled governments to enhance policy performance. Al-though design is gaining recogni-tion at the policy level, there are no studies that can ascertain indisput-ably the causal links between design and socio-economic development, isolating design from a wider context. Nevertheless, studies that have pro-vided evidence of positive correlations between design and its wider impacts on the economy and society should not be discouraged. The application of evaluation techniques for design needs to move beyond economic mea-sures to capture social and environ-mental change.

We conclude by reinforcing the complex nature of design evaluation, but highlight the need to overcome certain inefficiencies of current mea-suring practices. Improving our evalu-ation capabilities and understanding

ment, geographical distribution, and revenue, decision-makers will not be able to appreciate the scale of the design sector and its impact. The ef-fect of design on industry as a whole, as well as an analysis of the growth of the design sector, can be mea-sured over consecutive periods for comparable results. One-off studies are interesting but do not provide an insight into how the application of design resources is evolving. Research on this scale is costly but a number of orga-nizations have already developed the processes for conducting surveys of this magnitude, and the data gathered in such exercises has proved valuable.

Current practice in evaluat-ing design programs reveals a need to broaden focus toward outcomes and impact, rather than concentrate narrowly on delivery and operations. A wide variety of design support programs operate in Europe. To improve those currently in existence and to make way for future practice, we must improve the effectiveness of future evaluation. The parameters for how programs are measured are often constrained by frameworks deter-mined by the source of funding. This, although essential for the appropriate accountability of public funds, can

We conclude by reinforcing the complex nature of design evaluation, but highlight the need to overcome certain inefficiencies

of current measuring practices.