evaluation 2011: values and valuing in evaluation november 2-5, anaheim, california conjoint...

29
Evaluation 2011: Values and Valuing in Evaluation November 2-5, Anaheim, California Conjoint analysis for Contract Strategy Conjoint analysis for Contract Strategy for Culture Technology Enhancement for Culture Technology Enhancement Program in Korea Program in Korea Yun Jong Kim * , Uk Jung ** E-mail: [email protected], [email protected] *: KISTEP, Seoul, Korea, **: Dongguk University, Seoul, Korea

Upload: lauryn-critchett

Post on 29-Mar-2015

218 views

Category:

Documents


5 download

TRANSCRIPT

Page 1: Evaluation 2011: Values and Valuing in Evaluation November 2-5, Anaheim, California Conjoint analysis for Contract Strategy for Culture Technology Enhancement

Evaluation 2011: Values and Valuing in EvaluationNovember 2-5, Anaheim, California

Conjoint analysis for Contract Strategy for Culture Conjoint analysis for Contract Strategy for Culture Technology Enhancement Program in KoreaTechnology Enhancement Program in Korea

Yun Jong Kim*, Uk Jung **

E-mail: [email protected], [email protected]

*: KISTEP, Seoul, Korea, **: Dongguk University, Seoul, Korea

Page 2: Evaluation 2011: Values and Valuing in Evaluation November 2-5, Anaheim, California Conjoint analysis for Contract Strategy for Culture Technology Enhancement

【 Conjoint analysis for Contract Strategy for Culture Technology Enhancement Program in Korea 】

Overview (I)

As Culture Technology becomes one of the leading industries in its contribution to the economy, many countries are increasing national research investment through government-sponsored research projects.

Thus it becomes important to measure the importance that research participants attach to government-supported Culture Technology research project attributes.

Through some focus group discussions, a list of research project attributes for Culture

Technology was identified as important for participants for more successful project results.

This study uses conjoint analysis based on survey results to show that there is preferential difference in research project attributes for different affiliations of participants.

While the research was based on Korean experience, the research technique can be generalized to research policy designs in other countries.

2

Page 3: Evaluation 2011: Values and Valuing in Evaluation November 2-5, Anaheim, California Conjoint analysis for Contract Strategy for Culture Technology Enhancement

【 Conjoint analysis for Contract Strategy for Culture Technology Enhancement Program in Korea 】

Culture Technology in KoreaThe concept of CT in Korea was introduced in July 2001. It was immediately

adopted by the Korean government as one of the next six core technological engines for economic growth in the 21st century.

The six technology (6T), high value-added technology-intensive industries expected to lead

Korea's economic growth, were appointed for intensive support by the Korean government. Information Technology (IT), Biotechnology (BT), Nanotechnology (NT),

Space Technology (ST), Environmental Technology (ET), Culture Technology (CT)

CT is defined as “…technologies used in the value chain of culture content from the planning, commercialization, loading to media platforms, to distribution, and in a wide sense, complex technologies which are necessary for enhancing the added value to cultural products, including knowledge and knowhow from humanities and social science, design, and arts as well as science and engineering”

Similar concepts are Entertainment Technology and Creative Technology in other countries.

3

Page 4: Evaluation 2011: Values and Valuing in Evaluation November 2-5, Anaheim, California Conjoint analysis for Contract Strategy for Culture Technology Enhancement

【 Conjoint analysis for Contract Strategy for Culture Technology Enhancement Program in Korea 】

Culture Technology in Korea : Movies

4

Page 5: Evaluation 2011: Values and Valuing in Evaluation November 2-5, Anaheim, California Conjoint analysis for Contract Strategy for Culture Technology Enhancement

【 Conjoint analysis for Contract Strategy for Culture Technology Enhancement Program in Korea 】

Culture Technology in Korea : Movies

5

Page 6: Evaluation 2011: Values and Valuing in Evaluation November 2-5, Anaheim, California Conjoint analysis for Contract Strategy for Culture Technology Enhancement

【 Conjoint analysis for Contract Strategy for Culture Technology Enhancement Program in Korea 】

Culture Technology in Korea : Games

6

Page 7: Evaluation 2011: Values and Valuing in Evaluation November 2-5, Anaheim, California Conjoint analysis for Contract Strategy for Culture Technology Enhancement

【 Conjoint analysis for Contract Strategy for Culture Technology Enhancement Program in Korea 】

Culture Technology in Korea : Screen Golf

7

Page 8: Evaluation 2011: Values and Valuing in Evaluation November 2-5, Anaheim, California Conjoint analysis for Contract Strategy for Culture Technology Enhancement

【 Conjoint analysis for Contract Strategy for Culture Technology Enhancement Program in Korea 】

Culture Technologies in Many Countries

8

IRCAM (France)

ATR(Japan)

3C Research(England)

MIT Media Lab(America)

ITP(America)

Page 9: Evaluation 2011: Values and Valuing in Evaluation November 2-5, Anaheim, California Conjoint analysis for Contract Strategy for Culture Technology Enhancement

【 Conjoint analysis for Contract Strategy for Culture Technology Enhancement Program in Korea 】

Culture Technology

Especially, CT is the most typical technology field which is requiring the convergence and fusion of several technologies.

Apple’s DNA

It’s in Apple’s DNA that technology is not enough. It’s tech married with liberal arts and the

humanities. (2011. 3. 3. Keynote)

9

Page 10: Evaluation 2011: Values and Valuing in Evaluation November 2-5, Anaheim, California Conjoint analysis for Contract Strategy for Culture Technology Enhancement

【 Conjoint analysis for Contract Strategy for Culture Technology Enhancement Program in Korea 】

Culture Technology in Korea

Most government-supported CT research projects in Korea require different areas of knowledge to be put together and also require many different experts to work closely together.

Different areas of knowledge include humanities, Korean studies, engineering, marketing and

management and so on. As the results, the research collaboration is a key mechanism for

both knowledge production and diffusion in CT.

Thus, most CT research projects in Korea have basic attributes of inter-affiliated or

interdisciplinary research requirements for research collaboration.

The main objective of this study is to explore how the research participants feel about the collaboration issue with inter-affiliated or interdisciplinary research requirements.

For this purpose, this research uses conjoint analysis, which is appropriate in measuring the importance level or utility that research participants attach to the government-supported research project attributes including interaffiliated or interdisciplinary research requirements.

10

Page 11: Evaluation 2011: Values and Valuing in Evaluation November 2-5, Anaheim, California Conjoint analysis for Contract Strategy for Culture Technology Enhancement

【 Conjoint analysis for Contract Strategy for Culture Technology Enhancement Program in Korea 】

Conjoint Analysis (I)

Conjoint analysis is often used to understand how consumers develop preferences for products or services with multi-attribute levels in the marketing field.

Two basic assumptions are made in conjoint analysis.

1st, a product or service can be described as a combination of levels of a set of attributes.

2nd, these attribute levels determine the respondents’ overall judgment of the product or

service.

The attraction of using conjoint analysis is that it asks the respondents to make choices between products or services defined by a unique set of product/service attributes in a way resembling what they normally do-by trading off features, one against the other.

11

Page 12: Evaluation 2011: Values and Valuing in Evaluation November 2-5, Anaheim, California Conjoint analysis for Contract Strategy for Culture Technology Enhancement

【 Conjoint analysis for Contract Strategy for Culture Technology Enhancement Program in Korea 】

Conjoint Analysis (II)

Conjoint analysis produces two important results :

① Utility of attribute

[It] is a numerical expression of the value the respondents place in an attribute level.

It represents the relative “worth” of the attribute.

Low utility indicates less value; high utility indicates more value.

② Importance of attribute

[It] can be calculated by examining the difference between the lowest and highest utilities

across the levels of attributes.

12

Page 13: Evaluation 2011: Values and Valuing in Evaluation November 2-5, Anaheim, California Conjoint analysis for Contract Strategy for Culture Technology Enhancement

【 Conjoint analysis for Contract Strategy for Culture Technology Enhancement Program in Korea 】

Attributes in Conjoint Analysis Model

Attributes of inter-affiliated or interdisciplinary research requirements are our main interest in this study.

Inter-affiliated research means the research in which researchers from different affiliations

collaborate no matter whether researchers’ specialized research fields are same or not,

while interdisciplinary research means the research in which researchers with different

research fields of specialization work together.

To design the tradeoffs to represent the utility of inter-affiliated or interdisciplinary research requirements, we put into the government-supported research project attributes the research fund size and period as important attributes in this study.

13

Page 14: Evaluation 2011: Values and Valuing in Evaluation November 2-5, Anaheim, California Conjoint analysis for Contract Strategy for Culture Technology Enhancement

【 Conjoint analysis for Contract Strategy for Culture Technology Enhancement Program in Korea 】

Hypotheses

We will explore the utility of different levels of attributes and the importance of attributes the overall respondents perceived.

Then those utilities will be investigated in different participants group.

Using these results, we will try to address the existence of preferential difference among different participants group on the government-supported CT research program attributes with the following research hypotheses;

Hypothesis 1. CT research participants from different affiliations would have different preference on the levels of public funding size and period.

Hypothesis 2. CT research participants from different affiliations would have different preference on whether the inter-affiliated or interdisciplinary research was required.

14

Page 15: Evaluation 2011: Values and Valuing in Evaluation November 2-5, Anaheim, California Conjoint analysis for Contract Strategy for Culture Technology Enhancement

【 Conjoint analysis for Contract Strategy for Culture Technology Enhancement Program in Korea 】

Empirical Study (I)

We set up the research project attributes as Research Period, Research Fund Size, Inter-affiliated Research Requirement, and Interdisciplinary Research Requirement, in order to study suitable supporting policies for CT.

15

Factor Factor level

Research Period

Short Period (< 1 yr)

Medium Period (1~2 yrs )

Long Period (3+ yrs)

Research Fund Size

Small Fund (< 100 million won)

Medium Fund (100~ 299 million won)

Large Fund (300+ million won)

Inter-affiliated ResearchRequirement

With Inter-affiliation

Without Inter-affiliation

Interdisciplinary ResearchRequirement

With Interdisciplinary research

Without Interdisciplinary research

3 levels

3 levels

2 levels

2 levels

Page 16: Evaluation 2011: Values and Valuing in Evaluation November 2-5, Anaheim, California Conjoint analysis for Contract Strategy for Culture Technology Enhancement

【 Conjoint analysis for Contract Strategy for Culture Technology Enhancement Program in Korea 】

Empirical Study (II)

Despite a careful selection of factors, there were still too many (3322 = 36) possible profiles for the respondents.

The SPSS generated a parsimonious orthogonal array of 9 profiles.

16

ProfileResearch

PeriodResearch Fund Size

Inter-affiliated Research Requirement

Interdisciplinary Research Requirement

1 Medium Small No Yes

2 Long Small Yes No

3 Medium Large Yes No

4 Short Medium No No

5 Short Large Yes Yes

6 Short Small Yes Yes

7 Long Medium Yes Yes

8 Medium Medium Yes Yes

9 Long Large No Yes

Page 17: Evaluation 2011: Values and Valuing in Evaluation November 2-5, Anaheim, California Conjoint analysis for Contract Strategy for Culture Technology Enhancement

【 Conjoint analysis for Contract Strategy for Culture Technology Enhancement Program in Korea 】

Empirical Study (III)

It was decided it would be useful to study the utilities perceived by the following three different groups of affiliation participating in the research project; 1) industries, 2) academia, and 3) government-supported research institutes(GRIs).

There were a total of 128 respondents.

17

Affiliation Counts

Industries 30

Academia 76

GRIs 22

<Breakdown of affiliations>

Page 18: Evaluation 2011: Values and Valuing in Evaluation November 2-5, Anaheim, California Conjoint analysis for Contract Strategy for Culture Technology Enhancement

【 Conjoint analysis for Contract Strategy for Culture Technology Enhancement Program in Korea 】

Empirical Study (IV)

The relative importance levels of the various attributes are summarized.

Overall researchers generally consider the research period as the most important attributes, following by research fund size, inter-affiliation requirement, and interdisciplinary research requirement.

18

<Graph of averaged importance of various attributes>

Page 19: Evaluation 2011: Values and Valuing in Evaluation November 2-5, Anaheim, California Conjoint analysis for Contract Strategy for Culture Technology Enhancement

【 Conjoint analysis for Contract Strategy for Culture Technology Enhancement Program in Korea 】

Empirical Study (V)

One-way ANOVA analysis with a Least Significant Difference (LSD) Test at the 0.05 significance level was performed to compare the preferences of researchers from different affiliations.

19

ItemsOverall(n=128)

[A]Industry(

n=30)

[B]Academia

(n=76)

[C]GRI

(n=22)

ANOVA significance

levelConstant 4.88 5.15 4.81 4.78 AB, ACShort Period -0.37 0.14 -0.54 -0.45 ~Medium Period 0.17 0.30 0.19 -0.05 AB, ACLong Period 0.19 -0.44 0.36 0.50 ~Small Fund -0.85 -0.90 -0.70 -1.30 ~Medium Fund 0.64 0.46 0.62 0.95 ~Large Fund 0.21 0.44 0.08 0.35 ~With Inter-affiliation 0.27 -0.18 0.41 0.38 AB, ACWithout Inter-affiliation -0.27 0.18 -0.41 -0.38 AB, ACWith Interdisciplinary Research 0.08 -0.27 0.16 0.27 AB, AC

Without Interdisciplinary Research -0.08 0.27 -0.16 -0.27 AB, AC

<Comparison of means and ANOVA results for different affiliations>

Page 20: Evaluation 2011: Values and Valuing in Evaluation November 2-5, Anaheim, California Conjoint analysis for Contract Strategy for Culture Technology Enhancement

【 Conjoint analysis for Contract Strategy for Culture Technology Enhancement Program in Korea 】

Empirical Study (V)

One-way ANOVA analysis with a Least Significant Difference (LSD) Test at the 0.05 significance level was performed to compare the preferences of researchers from different affiliations.

20

ItemsOverall(n=128)

[A]Industry(

n=30)

[B]Academia

(n=76)

[C]GRI

(n=22)

ANOVA significance

levelConstant 4.88 5.15 4.81 4.78 AB, ACShort Period -0.37 0.14 -0.54 -0.45 ~Medium Period 0.17 0.30 0.19 -0.05 AB, ACLong Period 0.19 -0.44 0.36 0.50 ~Small Fund -0.85 -0.90 -0.70 -1.30 ~Medium Fund 0.64 0.46 0.62 0.95 ~Large Fund 0.21 0.44 0.08 0.35 ~With Inter-affiliation 0.27 -0.18 0.41 0.38 AB, ACWithout Inter-affiliation -0.27 0.18 -0.41 -0.38 AB, ACWith Interdisciplinary Research 0.08 -0.27 0.16 0.27 AB, AC

Without Interdisciplinary Research -0.08 0.27 -0.16 -0.27 AB, AC

<Comparison of means and ANOVA results for different affiliations>

No statistical difference at 0.05 significance level

Page 21: Evaluation 2011: Values and Valuing in Evaluation November 2-5, Anaheim, California Conjoint analysis for Contract Strategy for Culture Technology Enhancement

【 Conjoint analysis for Contract Strategy for Culture Technology Enhancement Program in Korea 】

Empirical Study (V)

One-way ANOVA analysis with a Least Significant Difference (LSD) Test at the 0.05 significance level was performed to compare the preferences of researchers from different affiliations.

21

ItemsOverall(n=128)

[A]Industry(

n=30)

[B]Academia

(n=76)

[C]GRI

(n=22)

ANOVA significance

levelConstant 4.88 5.15 4.81 4.78 AB, ACShort Period -0.37 0.14 -0.54 -0.45 ~Medium Period 0.17 0.30 0.19 -0.05 AB, ACLong Period 0.19 -0.44 0.36 0.50 ~Small Fund -0.85 -0.90 -0.70 -1.30 ~Medium Fund 0.64 0.46 0.62 0.95 ~Large Fund 0.21 0.44 0.08 0.35 ~With Inter-affiliation 0.27 -0.18 0.41 0.38 AB, ACWithout Inter-affiliation -0.27 0.18 -0.41 -0.38 AB, ACWith Interdisciplinary Research 0.08 -0.27 0.16 0.27 AB, AC

Without Interdisciplinary Research -0.08 0.27 -0.16 -0.27 AB, AC

<Comparison of means and ANOVA results for different affiliations>

difference in means at the 0.05 level for industry ([A])

and academia ([B])

Page 22: Evaluation 2011: Values and Valuing in Evaluation November 2-5, Anaheim, California Conjoint analysis for Contract Strategy for Culture Technology Enhancement

【 Conjoint analysis for Contract Strategy for Culture Technology Enhancement Program in Korea 】

Empirical Study (V)

The one-way ANOVA results can be interpreted as follows:

22

ItemsOverall(n=128)

[A]Industry(

n=30)

[B]Academia

(n=76)

[C]GRI

(n=22)

ANOVA significance

levelConstant 4.88 5.15 4.81 4.78 AB, ACShort Period -0.37 0.14 -0.54 -0.45 ~Medium Period 0.17 0.30 0.19 -0.05 AB, ACLong Period 0.19 -0.44 0.36 0.50 ~Small Fund -0.85 -0.90 -0.70 -1.30 ~Medium Fund 0.64 0.46 0.62 0.95 ~Large Fund 0.21 0.44 0.08 0.35 ~With Inter-affiliation 0.27 -0.18 0.41 0.38 AB, ACWithout Inter-affiliation -0.27 0.18 -0.41 -0.38 AB, ACWith Interdisciplinary Research 0.08 -0.27 0.16 0.27 AB, AC

Without Interdisciplinary Research -0.08 0.27 -0.16 -0.27 AB, AC

<Comparison of means and ANOVA results for different affiliations>

Compared to academia and GRI’s researchers, industry researchers had a higher constant value

Page 23: Evaluation 2011: Values and Valuing in Evaluation November 2-5, Anaheim, California Conjoint analysis for Contract Strategy for Culture Technology Enhancement

【 Conjoint analysis for Contract Strategy for Culture Technology Enhancement Program in Korea 】

Empirical Study (V)

The one-way ANOVA results can be interpreted as follows:

23

ItemsOverall(n=128)

[A]Industry(

n=30)

[B]Academia

(n=76)

[C]GRI

(n=22)

ANOVA significance

levelConstant 4.88 5.15 4.81 4.78 AB, ACShort Period -0.37 0.14 -0.54 -0.45 ~Medium Period 0.17 0.30 0.19 -0.05 AB, ACLong Period 0.19 -0.44 0.36 0.50 ~Small Fund -0.85 -0.90 -0.70 -1.30 ~Medium Fund 0.64 0.46 0.62 0.95 ~Large Fund 0.21 0.44 0.08 0.35 ~With Inter-affiliation 0.27 -0.18 0.41 0.38 AB, ACWithout Inter-affiliation -0.27 0.18 -0.41 -0.38 AB, ACWith Interdisciplinary Research 0.08 -0.27 0.16 0.27 AB, AC

Without Interdisciplinary Research -0.08 0.27 -0.16 -0.27 AB, AC

<Comparison of means and ANOVA results for different affiliations>

Industry researchers assigned higher utility to Medium research periods, whereas researchers from academia and GRIs assigned higher utility to Long research periods

Page 24: Evaluation 2011: Values and Valuing in Evaluation November 2-5, Anaheim, California Conjoint analysis for Contract Strategy for Culture Technology Enhancement

【 Conjoint analysis for Contract Strategy for Culture Technology Enhancement Program in Korea 】

Empirical Study (V)

The one-way ANOVA results can be interpreted as follows:

24

ItemsOverall(n=128)

[A]Industry(

n=30)

[B]Academia

(n=76)

[C]GRI

(n=22)

ANOVA significance

levelConstant 4.88 5.15 4.81 4.78 AB, ACShort Period -0.37 0.14 -0.54 -0.45 ~Medium Period 0.17 0.30 0.19 -0.05 AB, ACLong Period 0.19 -0.44 0.36 0.50 ~Small Fund -0.85 -0.90 -0.70 -1.30 ~Medium Fund 0.64 0.46 0.62 0.95 ~Large Fund 0.21 0.44 0.08 0.35 ~With Inter-affiliation 0.27 -0.18 0.41 0.38 AB, ACWithout Inter-affiliation -0.27 0.18 -0.41 -0.38 AB, ACWith Interdisciplinary Research 0.08 -0.27 0.16 0.27 AB, AC

Without Interdisciplinary Research -0.08 0.27 -0.16 -0.27 AB, AC

<Comparison of means and ANOVA results for different affiliations>

Industry researchers assigned lower utility to Inter-affiliation Research Requirements, whereas researchers from academia and GRIs assigned a higher utility.

Page 25: Evaluation 2011: Values and Valuing in Evaluation November 2-5, Anaheim, California Conjoint analysis for Contract Strategy for Culture Technology Enhancement

【 Conjoint analysis for Contract Strategy for Culture Technology Enhancement Program in Korea 】

Empirical Study (V)

The one-way ANOVA results can be interpreted as follows:

25

ItemsOverall(n=128)

[A]Industry(

n=30)

[B]Academia

(n=76)

[C]GRI

(n=22)

ANOVA significance

levelConstant 4.88 5.15 4.81 4.78 AB, ACShort Period -0.37 0.14 -0.54 -0.45 ~Medium Period 0.17 0.30 0.19 -0.05 AB, ACLong Period 0.19 -0.44 0.36 0.50 ~Small Fund -0.85 -0.90 -0.70 -1.30 ~Medium Fund 0.64 0.46 0.62 0.95 ~Large Fund 0.21 0.44 0.08 0.35 ~With Inter-affiliation 0.27 -0.18 0.41 0.38 AB, ACWithout Inter-affiliation -0.27 0.18 -0.41 -0.38 AB, ACWith Interdisciplinary Research 0.08 -0.27 0.16 0.27 AB, AC

Without Interdisciplinary Research -0.08 0.27 -0.16 -0.27 AB, AC

<Comparison of means and ANOVA results for different affiliations>

Industry researchers assigned lower utility to Interdisciplinary Research Requirements whereas researchers from academia and GRIs assigned higher utility.

Page 26: Evaluation 2011: Values and Valuing in Evaluation November 2-5, Anaheim, California Conjoint analysis for Contract Strategy for Culture Technology Enhancement

【 Conjoint analysis for Contract Strategy for Culture Technology Enhancement Program in Korea 】

Empirical Study (VI)

Hypothesis 1. CT research participants from different affiliations would have different preference on the levels of public funding size and period.

One can say that CT research participants from different affiliations have different preference on the levels of public funding period.

Industry researchers prefer medium research period to short or long period, whereas

researchers from academia and GRIs prefer long research periods.

However, as for research fund size, there are no significant differences in preference among research participants from different affiliations.

26

Page 27: Evaluation 2011: Values and Valuing in Evaluation November 2-5, Anaheim, California Conjoint analysis for Contract Strategy for Culture Technology Enhancement

【 Conjoint analysis for Contract Strategy for Culture Technology Enhancement Program in Korea 】

Empirical Study (VII)

Hypothesis 2. CT research participants from different affiliations would have different preference on whether the inter-affiliated or interdisciplinary research was required.

Industry researchers are reluctant to participate in inter-affiliated and interdisciplinary research projects, whereas researchers from academia and GRIs prefer them.

These barriers that industry researchers feel about the collaborations are discernible from the analysis regarding inter-affiliation and interdisciplinary research requirements.

27

Page 28: Evaluation 2011: Values and Valuing in Evaluation November 2-5, Anaheim, California Conjoint analysis for Contract Strategy for Culture Technology Enhancement

【 Conjoint analysis for Contract Strategy for Culture Technology Enhancement Program in Korea 】

Conclusion

This research illustrates the usefulness of conjoint analysis in determining the utility values of government-supported CT research project attributes.

The study demonstrates how evaluators can use this research technique to reveal and measure the hidden needs of participants.

The segmentation of participants into different groups according to affiliation has many practical applications.

With other research techniques such as cluster analysis and multi-dimensional scaling, using conjoint analysis offers extremely interesting academic and evaluating research opportunities.

28

Page 29: Evaluation 2011: Values and Valuing in Evaluation November 2-5, Anaheim, California Conjoint analysis for Contract Strategy for Culture Technology Enhancement

【 Conjoint analysis for Contract Strategy for Culture Technology Enhancement Program in Korea 】

Thank you very much!!