This article was downloaded by: [Northeastern University]On: 11 November 2014, At: 18:54Publisher: RoutledgeInforma Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer House,37-41 Mortimer Street, London W1T 3JH, UK

Creativity Research JournalPublication details, including instructions for authors and subscription information:http://www.tandfonline.com/loi/hcrj20

Application of the Rasch Model to the Measurement ofCreativity: The Creative Achievement QuestionnaireChia-Chi Wang a , Hsiao-Chi Ho a , Chih-Ling Cheng a & Ying-Yao Cheng aa National Sun Yat-Sen University , TaiwanPublished online: 26 Feb 2014.

To cite this article: Chia-Chi Wang , Hsiao-Chi Ho , Chih-Ling Cheng & Ying-Yao Cheng (2014) Application of the Rasch Modelto the Measurement of Creativity: The Creative Achievement Questionnaire, Creativity Research Journal, 26:1, 62-71, DOI:10.1080/10400419.2013.843347

To link to this article: http://dx.doi.org/10.1080/10400419.2013.843347

PLEASE SCROLL DOWN FOR ARTICLE

Taylor & Francis makes every effort to ensure the accuracy of all the information (the “Content”) containedin the publications on our platform. However, Taylor & Francis, our agents, and our licensors make norepresentations or warranties whatsoever as to the accuracy, completeness, or suitability for any purpose of theContent. Any opinions and views expressed in this publication are the opinions and views of the authors, andare not the views of or endorsed by Taylor & Francis. The accuracy of the Content should not be relied upon andshould be independently verified with primary sources of information. Taylor and Francis shall not be liable forany losses, actions, claims, proceedings, demands, costs, expenses, damages, and other liabilities whatsoeveror howsoever caused arising directly or indirectly in connection with, in relation to or arising out of the use ofthe Content.

This article may be used for research, teaching, and private study purposes. Any substantial or systematicreproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distribution in anyform to anyone is expressly forbidden. Terms & Conditions of access and use can be found at http://www.tandfonline.com/page/terms-and-conditions

CREATIVITY RESEARCH JOURNAL, 26(1), 62–71, 2014Copyright © Taylor & Francis Group, LLCISSN: 1040-0419 print/1532-6934 onlineDOI: 10.1080/10400419.2013.843347

Application of the Rasch Model to the Measurement of Creativity: The Creative Achievement Questionnaire

Chia-Chi Wang, Hsiao-Chi Ho, Chih-Ling Cheng, and Ying-Yao ChengNational Sun Yat-Sen University, Taiwan

This study was designed to provide multiple sources of evidence of the validity of the Creative Achievement Questionnaire (CAQ) and to clarify the hierarchy of creative achievement using Rasch analyses. A total of 905 Taiwanese participants (345 men and 558 women) completed the CAQ online. The Rasch model was used to assess model–data fit. A differential item functioning (DIF) analysis was conducted to assess the consistency of the ratings provided by males and females. The results revealed that the 10 dimensions of the CAQ showed good model–data fit and supported the two-factor classification (arts and sciences). Additionally, several items from each domain exhibited substantial DIF across genders. Moreover, with the exception of dance, cre-ative writing, architectural design, theater and film, and culinary arts, items in the 10 domains were hierarchically ordered. Suggestions for future research to revise the CAQ were proposed.

Creative achievement can be defined as the sum of creative products that are original, valuable, and pro-duced by an individual over the course of his or her lifetime (Carson, Peterson, & Higgins, 2005; Helson & Pals, 2000). Common methods for measuring creative achievement include the use of verifiable accomplish-ments or honors, ratings of existing creative products by experts, and self-report achievement inventories (SAIs) as markers. Although time-consuming, SAIs are regarded as more comprehensive than other meth-ods because they can be designed to measure multiple dimensions of achievement.

A number of self-report inventories have been designed to assess creative achievement: (a) the Creative Behavior Inventory developed by Hocevar (1979) is a list of 90 creative actions in the domains of literature, music, miscellaneous, math/science, art, and performing arts; (b) the Creative Achievement Scale (Ludwig, 1992) mea-sures remarkable creative achievements in terms of per-sonal attributes, product qualities, and sociocultural factors; (c) the Creative Achievement Questionnaire (CAQ) developed by Carson et al. (2005) uses

rank-ordered items to evaluate the level of creative achievement across 10 domains including visual arts, music, dance, creative writing, architectural design, humor, invention, scientific inquiry, theater and film, and culinary arts; and (d) the Biographical Inventory of Creative Behaviours developed by Batey and Furnham (2008), evaluates creative achievements in terms of 34 activities (e.g., writing a short story, producing one’s own Web site, publishing research, designing and planting a garden, composing a piece of music).

The CAQ is the most frequently used self-report inven-tory of creative achievement and has been cited 49 times in the Social Sciences Citation Index, 45 times in PsycINFO, and 124 times in Google Scholar between December 2005 and December 2012 (e.g., Batey, 2012; Reiter-Palmon, Robinson-Morral, & Kaufman, 2012; Simonton, 2012). The CAQ was based on a comprehen-sive review of the creative accomplishment areas identi-fied by previous research and included both arts and science domains (Carson et al., 2005; Colangelo, Kerr, Hallowell, Huesman, & Gaeth, 1992; Hocevar, 1979; MacKinnon, 1962; Taylor & Ellison, 1967; Torrance, 1972). Furthermore, the CAQ was the first measure of creative achievement to include the culinary arts as a domain of creative achievement. Although Carson and colleagues did not clearly explain why the culinary arts

Correspondence should be sent to Ying-Yao Cheng, Institute of Education, National Sun Yat-sen University, 70 Lienhai Rd., Kaohsiung 80424, Taiwan. E-mail: chengyy@mail.nsysu.edu.tw

Dow

nloa

ded

by [

Nor

thea

ster

n U

nive

rsity

] at

18:

54 1

1 N

ovem

ber

2014

THE CREATIVE ACHIEVEMENT QUESTIONNAIRE 63

was regarded as an area for creative achievement, their novel idea was supported by the findings of Horng and Hu (2008), which showed that culinary creativity involves food, art, esthetics, and sciences such as physics, chemis-try, and biology.

Despite the novel aspects of the CAQ, the instru-ment has several limitations with regard to validity. First, the CAQ was developed using a homogeneous sample of relatively gifted university students; thus, the instrument may not be widely representative and have an inference bias. Second, the CAQ is a domain-specific tool used to assess 10 areas of creative achievement; however, the total score for the 10 domains is used to indicate overall creative performance, which appears to violate domain specificity, one of the major assump-tions of the CAQ. Third, Carson et al. (2005) provided evidence for construct validity by extracting common factors using an exploratory factor analysis. As the data analysis was conducted on responses from participants and not on theory, the resulting factor structure might be inconsistent. Fourth, although the CAQ relies on a hierarchy of items, the hierarchy of creative achieve-ments was based on consultations with only two domain experts, which suggests a lack of objectivity. Moreover, like most of self-report creativity achievement measures built upon classical test theory (CTT), the original test scores of CAQ are unsuitable for data analysis since they are not based on interval scales and have biases resulting from being dependent on test sample (Wang, 2004).

In recent years, Rasch models have been widely used to test the quality of questionnaires in fields such as human science, education, medical science, manage-ment (Wang, 2004), and creativity research (Teo & Waugh, 2010; Wang & Cheng, 2000). Rasch models may overcome some of the limitations of psychological test-ing associated with CTT; for example, when the observed responses fit well with the model, the ordinal scale can be converted into an interval scale using a Rasch analy-sis. Additionally, because Rasch measurement is based on theoretical foundations, it can avoid the inconsistent factor structure arising from exploratory factor analy-sis. Furthermore, Rasch analysis not only examines the-oretical constructs, but also provides information regarding item hierarchies and latent traits. Overall, the quality of an assessment tool can be tested using the multiple sources of evidence offered by the Rasch measurement approach.

This study used the CAQ to demonstrate the use of the Rasch model to validate a creativity measure. This study also assessed the content validity, structural valid-ity, generalizability, substantive validity, and interpret-ability (Messick, 1994, 1995a, 1995b; Wolfe & Smith, 2007) of the CAQ using a Rasch analysis, and suggestions for CAQ revisions were made.

METHOD

Participants

Data collection proceeded in two phases involving two groups of participants. In the first phase, responses were obtained from 905 participants (92.6% of deliveries), including 345 men, 558 women, and 2 who did not spec-ify their sex. Subject ages ranged from 14 to 78 years (M = 23.32 years, SD = 5.34). Participants were asked to complete the CAQ on a free Web site and their Internet protocol (IP) addresses were recorded so repeated responses could be deleted from the database.

The CAQ asked participants to specify in which of 13 creative areas they perceived themselves to be more talented or competent than the average person. Of the 905 participants, 31.93% (n = 289) chose visual arts (painting, sculpture), 37.46% (n = 339) music, 11.16% (n = 101) dance, 19.12% (n = 173) individual sports (ten-nis, golf), 23.87% (n = 216) team sports, 10.83% (n = 98) architectural design, 10.17% (n = 92) entrepreneurship, 30.61% (n = 277) creative writing, 37.35% (n = 338) humor, 11.49% (n = 104) inventions, 18.45% (n = 167) scientific inquiry, 25.64% (n = 232) theater and film, and 24.64% (n = 223) culinary arts.

As the first phase of the study showed that the test results for five domains did not fit the theoretical hier-archy, in the second phase, one expert in each of those domains was invited to rerank the CAQ domain order and provide suggestions for further investigations of the order of items. Several criteria were used to select the domain experts: more than 10 years of professional experience in the specific domain, receipt of awards in regional or national competitions, or certification from national institutes. In the dance domain, Expert A was a female creative director of a dance company and a professional dance instructor who had won several regional and national awards. In the architectural design domain, Expert B was a female studying at an architec-tural institute who specialized in architectural design, urban design, green architecture, and indoor design. She had passed the national examinations for architec-ture in Taiwan. In the domain of creative writing, Expert C was a female editor of a local art magazine. In the theater and film domain, Expert D was a female professional who had worked in an advertising com-pany for more than 10 years. In the culinary arts domain, Expert E was a female Chinese cooking instructor and certified chef.

Instrument

This study used the CAQ as the instrument. Regarding back translation process of the CAQ, two bilingual experts were asked to translate the English version of the

Dow

nloa

ded

by [

Nor

thea

ster

n U

nive

rsity

] at

18:

54 1

1 N

ovem

ber

2014

64 WANG ET AL.

and film were categorized as arts, whereas inventions, scientific discoveries, and the culinary arts were cate-gorized as sciences. Using a multidimensional Rasch analysis, this study examined whether the two-factor construct of the CAQ fit the model’s expectations and calculated the person separation reliability and correlations.

With respect to generalizability, differential item functioning (DIF; Holland & Wainer, 1993) analyses were conducted across genders. A difference of 0.5 logits in the overall difficulty of items across groups was regarded as a substantial DIF (Wang, 2008). A multidi-mensional Rasch analysis was conducted to obtain an accurate estimate of the correlations among the 10 sub-scales and to increase person reliability (Cheng, Wang, & Ho, 2009). Finally, to evaluate the substantive aspect of validity, the hypothesized item hierarchy was confirmed by comparing it with the empirically derived hierarchy of item calibrations using Spearman rank correlations (Wilson, 2005).

RESULTS

Data Analysis

Content and structural validity. The Rasch model was used to examine the unidimensionality of each of the 10 CAQ subscales. The results revealed that all items in each domain had acceptable infit MNSQs (Table 1). Data from each domain fit the Rasch model well, indicating that the items in each subscale meas-ured a unidimensional construct. The multidimen-sional Rasch analysis conducted within the 10 domains of the CAQ indicated that not all items in each domain fit the Rasch model well (Table 1). Furthermore, multi-dimensional Rasch analysis conducted on the two-fac-tor construct (arts and sciences) derived from the 10 domains exhibited good model–data fit, confirming the two-factor structure of creative achievement (Table 1). Table 2 shows the corre lations among the 10 domains.

Generalizability validity. DIF analyses were con-ducted separately for each of the 10 CAQ subscales. All items in each domain, with the exception of theater and film, were associated with substantial DIF between the genders. A total of nine items in all domains exhibited substantial DIF for men and women, indicating that the meaning of these items differed for men and women. The person separation reliabilities of the 10 subscales, shown in Table 2, ranged from .63 to .77. Of these sub-scales, the person separation reliability in the dance domain (.63) was lower than were those in the other

questionnaire into Chinese, and one expert ensured con-sistency between the versions.

The present study involved Parts One and Two of the CAQ, yielding a total of 93 items. The CAQ is a self-report tool with rank-ordered items designed to evaluate the level of achievement across 10 domains of creativity. Items in Part One measure talents in 13 areas, and participants were asked to select areas in which they perceived themselves to be more talented or competent than the average person. Items in Part Two evaluate concrete achievements in the 10 domains of visual arts, music, dance, creative writing, architectural design, humor, inventions, scientific inquiry, theater and film, and culinary arts. Each domain is assessed by asking participants to rank eight items from lowest to highest.

Analyses

The Rasch model (Rasch, 1960) consists of two primary parameters: person’s (n) ability (θn) and item (i) difficulty (δi). When a person n responds to item i, the probability of that person being correct on that item is:

)

)1exp(

1 exp(n i

nin i

Pθ δ

θ δ−

=+ −

, (1)

where Pni1 is the probability of person n scoring 1 on item i. Additionally, θn, which is the latent trait level of person n, refers to the construct that is the target of the measure-ment (e.g., personality, attitudes, interests, values, perfor-mance), and δj is the overall difficulty of item i (e.g., difficulty or threshold value). Objective responses refer to dichotomous scores (e.g., success or failure, agree or dis-agree), and Pni1 and Pni0 denote the probabilities of per-son n scoring 1 and 0, respectively, on item i. Thus, Pni1 + Pni0 = 1.

The Rasch model was employed because each item of the CAQ has binary response categories (yes/no). Parameters in the Rasch model were estimated using the ConQuest computer program (Wu, Adams, & Wilson, 2007).

With regard to content and structural validity, the Rasch model was used to examine the fit of each item and the unidimensionality of each domain. When data fit the expectation generated by the model, the infit (weighted) mean square error (MNSQ) had an expected value of unity. MNSQ values between 0.6 and 1.4 were set as the criteria for a reasonably good model–data fit (Wright & Linacre, 1994). Additionally, the two-factor structure identified by Carson et al. (2005) was used to classify creative achievements in terms of arts and sciences. The domains of visual arts, music, dance, creative writing, humor, and theater

Dow

nloa

ded

by [

Nor

thea

ster

n U

nive

rsity

] at

18:

54 1

1 N

ovem

ber

2014

THE CREATIVE ACHIEVEMENT QUESTIONNAIRE 65

TABLE 1Estimates of Item Difficulty and Goodness-of-Fit Values for the Arts and Sciences by Multidimensional Rasch Analysis for Each Domain by

Unidimensional and Multidimensional Rasch Analyses

Factor Domain Item

Arts & Science Factor by Multidimensional Rasch.

Each Domain by Multidimensional Rasch

Each Domain by Unidimensional Rasch

Estimate Infit MNSQ Estimate Infit MNSQ Estimate Infit MNSQ

Factor 1 Arts Visual arts I1 –3.04 0.96 –2.38 0.97 –2.41 1.01I2 –3.25 1.05 –2.65 1.16 –2.68 1.13I3 –2.29 1.00 –1.41 0.98 –1.43 0.99I4 –1.75 1.03 –0.71 0.95 –0.73 0.90I5 –1.09 1.03 0.12 1.01 0.12 0.90I6 0.54 1.02 2.09 1.21 2.12 1.02I7 0.44 0.97 1.97 1.10 2.00 1.01I8 1.33 1.03 2.97 1.20 3.03 1.07

Music I1 –3.62 1.05 –4.76 1.02 –5.38 0.99I2 –2.71 1.05 –3.58 0.99 –4.05 0.96I3 –0.07 1.02 –0.35 1.07 –0.38 1.01I4 –0.53 1.03 –0.88 1.10 –0.98 1.02I5 2.24 1.00 2.15 0.98 2.41 1.11I6 1.61 1.04 1.51 1.00 1.69 1.04I7 3.30 1.09 3.14 0.91 3.57 1.25I8 2.88 1.06 2.77 0.98 3.12 1.20

Dance I1 –2.03 1.07 –4.33 1.13 –5.01 1.11I2 1.17 1.03 0.49 1.33 0.56 1.03I3 0.16 1.00 –1.00 1.05 –1.14 0.83I4 0.05 1.01 –1.16 0.98 –1.33 0.79I5 2.15 1.05 1.84 1.12 2.12 1.28I6 1.17 1.02 0.48 1.13 0.56 0.85I7 2.07 1.05 1.72 1.23 1.99 1.13I8 2.25 1.05 1.96 1.15 2.25 1.27

Creative writing I1 –3.11 0.97 –2.92 1.08 –3.02 1.09I2 –3.00 1.00 –2.78 1.01 –2.87 0.96I3 –1.77 0.98 –1.20 0.99 –1.22 0.97I4 –0.47 1.04 0.44 1.14 0.45 1.10I5 1.42 1.02 2.58 1.10 2.68 1.04I6 –0.07 1.03 0.92 1.08 0.94 0.99I7 –0.22 1.00 0.74 1.00 0.76 0.91I8 1.07 0.98 2.23 1.10 2.28 0.98

Humor I1 –4.08 1.03 –5.08 1.00 –5.46 0.98I2 –2.61 1.08 –3.16 1.00 –3.37 0.95I3 –2.11 1.06 –2.51 1.00 –2.66 0.97I4 –0.30 1.05 –0.25 1.10 –0.23 1.04I5 2.00 1.05 2.36 1.16 2.53 1.10I6 1.92 1.03 2.29 1.15 2.45 1.11I7 3.08 1.08 3.53 1.21 3.70 1.24I8 2.46 1.07 2.83 1.16 3.04 1.17

Theater/film I1 –2.09 1.01 –3.03 1.08 –3.72 1.11I2 –1.34 0.99 –2.17 1.04 –2.64 0.96I3 1.78 1.03 1.32 1.15 1.62 1.18I4 0.11 0.95 –0.53 0.97 –0.61 0.91I5 1.17 1.00 0.65 1.05 0.82 1.01I6 0.94 0.98 0.39 1.00 0.51 0.94I7 1.73 1.01 1.26 1.09 1.54 1.00I8 2.46 1.02 2.11 1.25 2.47 1.26

Factor 2 Science Invention I1 –3.54 0.94 –3.98 1.06 –4.17 1.04I2 –3.15 0.96 –3.48 1.08 –3.63 1.05I3 –1.59 0.95 –1.44 0.97 –1.49 0.89I4 0.69 0.97 1.34 1.13 1.39 1.11I5 –0.89 0.96 –0.55 1.00 –0.57 0.89I6 1.76 1.03 2.53 1.11 2.65 1.13I7 1.74 0.96 2.53 1.12 2.64 1.17

(Continued)

Dow

nloa

ded

by [

Nor

thea

ster

n U

nive

rsity

] at

18:

54 1

1 N

ovem

ber

2014

66 WANG ET AL.

indicate less likelihood that an individual reported a cre-ative achievement for that item.

Overall, the items in the following domains were higher than participants’ creativity: visual arts (M = –1.96, SD = 1.89), music (M = –4.05, SD = 2.22), dance (M = –6.23, SD = 3.16), architectural design (M = –6.60, SD = 2.80), creative writing (M = –2.47, SD = 1.97), humor (M = –3.52, SD = 2.16), invention (M = –4.24, SD = 2.50), scientific inquiry (M = –4.19, SD = 2.52), theater and film (M = –4.61, SD = 2.32),

domains, indicating that items in the dance domain were less consistent than were items in other domains.

Interpretability validity. Figures 1–10 show the per-son measures and item difficulty for each CAQ domain. Each X on the left side of the figure denotes a person with a creative ability in that domain. The digit on the right denotes the item number. Positive values indicate higher levels of creativity achieved by that individual, and more positive values for an item (i.e., higher levels)

TABLE 2 Reliabilities and Correlations Among 10 Domains

1 2 3 4 5 6 7 8 9 10

1. Visual arts (.72) 2. Music 0.42 (.67) 3. Dance 0.31 0.38 (.63) 4. Architectural design 0.57 0.47 0.69 (.69) 5. Creative writing 0.41 0.45 0.37 0.30 (.77) 6. Humor 0.24 0.25 0.32 0.34 0.38 (.73) 7. Inventions 0.31 0.19 0.18 0.55 0.26 0.47 (.77) 8. Scientific discovery 0.11 0.16 0.09 0.40 0.13 0.35 0.72 (.73) 9. Theater/film 0.44 0.46 0.54 0.68 0.79 0.49 0.34 0.18 (.73)10. Culinary arts 0.54 0.61 0.66 0.59 0.65 0.39 0.56 0.38 0.55 (.69)

Note. ( ) = person separation reliability.

TABLE 1 (Continued)

Factor Domain Item

Arts & Science Factor by Multidimensional Rasch.

Each Domain by Multidimensional Rasch

Each Domain by Unidimensional Rasch

Estimate Infit MNSQ Estimate Infit MNSQ Estimate Infit MNSQ

I8 2.21 1.02 3.05 1.20 3.17 1.21Scientific discovery I1 –3.42 1.01 –3.66 1.07 –3.98 1.04

I2 –3.23 0.98 –3.41 1.03 –3.71 1.06I3 –1.13 0.98 –0.81 1.07 –0.83 1.01I4 0.22 0.92 0.83 1.10 0.96 0.92I5 0.07 0.93 0.65 1.17 0.76 0.93I6 1.97 0.99 3.04 a1.47 3.11 1.32I7 0.54 0.84 1.21 0.99 1.38 0.85I8 1.30 0.93 2.15 1.26 2.32 1.08

Culinary arts I1 –2.82 1.26 –3.00 1.05 –3.85 1.08I2 –2.68 1.25 –2.89 1.04 –2.57 1.04I3 2.35 1.09 1.25 0.73 –2.44 0.97I4 1.35 1.06 0.50 0.84 2.21 1.06I5 2.21 1.05 1.15 0.75 1.28 1.03I6 2.08 1.10 1.06 0.78 2.07 1.05I7 1.43 1.06 0.56 0.85 1.95 1.05I8 2.53 1.11 1.37 0.71 1.34 1.02

Other Architectural design I1 — — –3.93 1.34 –4.17 1.14I2 — — –1.91 1.16 –2.04 0.94I3 — — 0.58 1.23 0.62 1.07I4 — — 1.14 a1.48 1.22 1.33I5 — — 1.14 a1.46 1.22 1.32I6 — — 1.02 1.36 1.09 1.25I7 — — 0.79 1.22 0.84 1.14I8 — — 1.16 a1.40 1.22 1.37

Note. a = poor model–data fit.—= no values.

Dow

nloa

ded

by [

Nor

thea

ster

n U

nive

rsity

] at

18:

54 1

1 N

ovem

ber

2014

THE CREATIVE ACHIEVEMENT QUESTIONNAIRE 67

Substantive validity. Each domain in the CAQ was composed of eight items arranged in order of low-est to highest level of creative achievement. The Spearman rank correlation of item estimates was used to examine calibration invariance between the expected

and culinary arts (M = –3.25, SD = 1.10). The average cre-ativity of participants was lower than their creative achievement in regard to items in each domain, indicat-ing that most participants had low levels of creative achievement.

FIGURE 1 Visual arts.

FIGURE 2 Music.

FIGURE 3 Dance.

FIGURE 4 Architectural design.

Dow

nloa

ded

by [

Nor

thea

ster

n U

nive

rsity

] at

18:

54 1

1 N

ovem

ber

2014

68 WANG ET AL.

(original) and the empirically derived item hierarchy. Correlations between the two sets of rankings for the visual arts, music, humor, inventions, and scientific discovery domains were greater than .91 (p < 0.01).

Correlations between the two sets of rankings for the dance, architectural design, creative writing, theater/film, and culinary arts domains ranged from .59 to .88 (p > 0.01; Table 3), suggesting that the relative

FIGURE 6 Humor.

FIGURE 5 Creative writing. FIGURE 7 Inventions.

FIGURE 8 Scientific discovery.

Dow

nloa

ded

by [

Nor

thea

ster

n U

nive

rsity

] at

18:

54 1

1 N

ovem

ber

2014

THE CREATIVE ACHIEVEMENT QUESTIONNAIRE 69

domain, the correlations between Expert B’s rankings and the empirically derived data (r = .59, p > 0.01) or those of the original hypothesis (r = .38, p > 0.01) were not significant. Similarly, the rankings provided by Expert C for the creative writing domain were not sig-nificantly correlated with the empirically derived data (r = .59, p > 0.01) or those generated by the original hypothesis (r = .38, p > 0.01). The rankings provided by Expert D for the theater and film domain were not sig-nificantly correlated with the empirically derived data (r = .50, p > 0.01) or those generated by the original hypothesis (r = .10, p > 0.01). Finally, the rankings of Expert E for the culinary arts domain were not signifi-cantly correlated with the empirically derived data (r = .10, p > 0.01) or those generated by the original hypothesis (r = .57, p > 0.01).

difficulty of the items in these five subscales required modification.

Experts’ Ranking

To clarify the discrepancy in the item hierarchies of the five domains, an expert in each of the domains was asked to rank the items. Next, the rankings provided by the domain experts, the empirically derived data, and the data generated by the original hypothesis were com-pared using Spearman rank correlations. In the dance domain, the correlations between Expert A’s rankings and the empirically derived data were significant (r = .91, p < 0.01); however, the correlations between Expert A’s rankings and those generated by the original hypothesis were not (r = .79, p > 0.01). In the architectural design

FIGURE 9 Theater/film. FIGURE 10 Culinary arts.

TABLE 3Spearman Rank Correlations Between Expected and Empirically Derived Item Hierarchies

Domain Visual Arts Music Dance Architectural Design Creative Writing

RankExpected order .95(**) .93(**) .88 .59 .83

Domain Humor Invention Scientific Discovery Theater/Film Culinary Arts

RankExpected order .95(**) .95(**) .91(**) .74 .67

Note.**Correlation is significant at the 0.01 level (two-tailed).

Dow

nloa

ded

by [

Nor

thea

ster

n U

nive

rsity

] at

18:

54 1

1 N

ovem

ber

2014

70 WANG ET AL.

subjects with the same ability. This finding indicated that the generalizability of the CAQ should be improved.

Additionally, the results for the architectural design and culinary arts domains indicate that several overlap-ping items should be reviewed (see Figures 4 and 10). In terms of the architectural design domain, the creative achievements described in items 3–8 were difficult for par-ticipants to achieve. In other words, the ranking of items in this domain appeared to be inconsistent with the achievements of participants. This pattern was also observed in the culinary arts domain. It is likely that few participants in the study had creative abilities related to architectural design or culinary arts; however, the repre-sentative sample was too insufficient to recheck the item rankings. Thus, the ranking of items in these two sub-scales should be rechecked using participants with the appropriate professional expertise. Finally, the Rasch analysis revealed inconsistencies in the item hierarchy in the dance, architectural design, creative writing, theater and film, and culinary arts domains. To clarify the factors underlying this discrepancy, an expert in each of the five domains was asked to rank the items. The results revealed that the hierarchical ranking provided by the five experts were more consistent with the order of the empirical data than that in the theoretical hierarchy. This finding sug-gests that follow-up studies are necessary to reconfirm item rankings in each CAQ domain. Furthermore, the CAQ should be reexamined with respect to the number and nature of items in each domain. To establish the lev-els of creative achievement in different domains and to validate and improve the quality of the CAQ, in-depth interviews with focus groups and with experts in each domain and tests of the validity of the instrument in a larger and more representative sample than that used in the present study are necessary.

Although the value of the Rasch model in obtaining objective psychometric tests has been increasingly recog-nized, the model is rarely applied to creativity measurement. Using the CAQ (Carson et al. 2005) as an example, this study has demonstrated that the Rasch model can be used to validate and improve an existing creativity measurement tool that was developed using CTT and can confirm the item hierarchy and level of creativity through responses to indi-vidual items. We hope that our study stimulates future investigations that apply the Rasch model to improve the quality and objectivity of creativity assessment tools.

REFERENCES

Batey, M. (2012). The measurement of creativity: From definitional consensus to the introduction of a new heuristic framework. Creativity Research Journal, 24, 55–65. doi: 10.1080/10400419.2012.649181

Batey, M., & Furnham, A. (2008). The relationship between measures of creativity and schizotypy. Personality and Individual Differences, 45, 816–821. doi: 10.1016/j.paid.2008.08.014

DISCUSSION AND CONCLUSIONS

The CAQ is a practical and frequently used tool for mea-suring and examining various components of creative achievement. However, investigations that ignore the assumptions and limitations of the CAQ may produce biased results. This study examined the validity of the CAQ using Rasch models to overcome these potential biases.

The first phase of the study involved 905 participants aged between 14 and 78 years. This large and heteroge-neous sample was designed to avoid the limitations arising from a homogeneous sample, such as that used in the study conducted by Carson et al. (2005), in which most partici-pants were relatively gifted students attending Harvard University. The results of our study revealed that the aver-age creative achievement of participants was lower than the item difficulty and that most participants achieved lower levels of creative achievement than expected. The majority of respondents fell into the less-creative achieve-ment group, with only a few in the high-creative achieve-ment group (Carson et al., 2005; Eysenck, 1995). Similar results were reported by Silvia, Kaufman, and Pretz (2009).

Second, Rasch analysis was utilized to validate the construct components of the CAQ and provided multiple evidence of validity. Previous studies have shown that cre-ative ability differs across domains and is domain-specific (e.g., arts vs. sciences; Carson et al., 2005; Gruber & Wallace, 1999; Policastro & Gardner, 1999). Ivcevic (2009) also reported that most creativity is domain- specific. The Rasch analyses used in this study demon strated that all items in each domain fit their respective construct well and supported the two-factor classi fication (arts and sci-ences) proposed by Carson et al. (2005). Moreover, the CAQ was transformed into an objective interval scale using the Rasch analysis, which has important implica-tions for future research. For instance, using the CAQ to assess creative achievement allows each domain to be con-sidered as an independent category, and several similar domains may be integrated into a more general domain (e.g., inventions, scientific discoveries, and culinary arts can be combined into a single category of creative scien-tific achievement). Additionally, a hierarchical relation-ship may exist among different domains, allowing a test of the domain hierarchy theory (Simonton, 2009), which states that one dimension of the variability may coordi-nate the disconnected data points of various domains.

Third, this study validated the generalizability of the CAQ using DIF analysis. In theory, a single instrument designed to measure a given latent trait or ability should be applicable to all groups. The DIF reflects the degree to which a given item holds a different meaning for partici-pants with the same ability belonging to different groups. The results of this study revealed that the items in most of the CAQ domains (with the exception of theater and film) may be interpreted differently by male and female

Dow

nloa

ded

by [

Nor

thea

ster

n U

nive

rsity

] at

18:

54 1

1 N

ovem

ber

2014

THE CREATIVE ACHIEVEMENT QUESTIONNAIRE 71

(Ed.), Handbook of creativity (pp. 213–255), New York, NY: Cambridge University Press.

Rasch, G. (1960). Probabilistic models for some intelligence and attain-ment tests. Copenhagen: Institute of Educational Research.

Reiter-Palmon, R., Robinson-Morral, E. J., & Kaufman, J. C. (2012). Evaluation of self-perceptions of creativity: Is it a useful criterion? Creativity Research Journal, 24, 107–114. doi: 10.1080/10400419.2012.676980

Silvia, P. J., Kaufman, J. C., & Pretz, J. E. (2009). Is creativity domain-specific? Latent class models of creative accomplishments and cre-ative self-descriptions. Psychology of Aesthetics, Creativity, and the Arts, 2, 139–148. doi: 10.1037/a0014940

Simonton, D. K. (2009). Varieties of (scientific) creativity: A hierarchi-cal model of disposition, development, and achievement. Perspectives on Psychological Science, 4, 441–452.

Simonton, D. K. (2012). Taking the US Patent Office criteria seriously: A quantitative three-criterion creativity definition and its implica-tions. Creativity Research Journal, 24, 97–106. doi: 10.1080/10400419.2012.676974

Taylor, C. W., & Ellison, R. L. (1967). Biographical predictors of scien-tific performance. Science, 155, 1075–1079. doi: 10.1126/science.155.3766.1075

Teo, L. M. C., & Waugh, R. F. (2010). A Rasch measure of fostering creativity. Creativity Research Journal, 22, 206–218. doi: 10.1080/10400419.2010.481534

Torrance, E. P. (1972). Career patterns and peak creative achievements of creative high school students twelve years later. Gifted Child Quarterly, 16, 75–88.

Wang, W. C. (2004). Rasch measurement theory and application in edu-cation and psychology. Journal of Education and Psychology, 27, 637–694.

Wang, W. C. (2008). Assessment of differential item functioning. Journal of Applied Measurement, 9, 1–22.

Wang, W. C., & Cheng, Y. Y. (2000). Development and item response analysis of the creativity development inventory. Psychological Testing, 47, 153–173.

Wilson, M. (2005). Constructing measures: An item response modeling approach. Mahwah, NJ: Lawrence Erlbaum Associates.

Wolfe, E. W., & Smith, E. V. (2007). Instrument development tools and activities for measure validation using Rasch models: Part II-Validation activities. Journal of Applied Measurement, 8, 204−233.

Wright, B. D., & Linacre, J. M. (1994). Reasonable mean-square fit val-ues. Rasch Measurement Transactions, 8, 370.

Wu, M. L., Adams, R. J., & Wilson, M. R. (2007). ConQuest [Computer software and manual]. Camberwell, Australia: Australian Council for Educational Research.

Carson, S. H., Peterson, J. B., & Higgins, D. M. (2005). Reliability, valid-ity, and factor structure of the creative achievement questionnaire. Creativity Research Journal, 17, 37–50. doi: 10.1207/s15326934crj1701_4

Cheng, Y. Y., Wang, W. C., & Ho, Y. H. (2009). Multidimensional Rasch analysis of a psychological test with multiple subtests: A sta-tistical solution for the bandwidth–fidelity dilemma. Educational and Psychological Measurement, 69, 369–388. doi: 10.1177/0013164408323241

Colangelo, N., Kerr, B., Hallowell, K., Huesman, R., & Gaeth, J. (1992). The Iowa Inventiveness Inventory: Toward a measure of mechanical inventiveness. Creativity Research Journal, 5, 157–163. doi: 10.1080/10400419209534429

Eysenck, H. J. (1995). Genius: The natural history of creativity. Cambridge, UK: Cambridge University Press.

Gruber, H. E., & Wallace, D. B. (1999). The case study method and evolving systems approach for understanding unique creative people at work. In R. J. Sternberg (Ed.), Handbook of creativity (pp. 93–115). Cambridge, UK: Cambridge University Press.

Helson, R., & Pals, J. L. (2000). Creative potential, creative achieve-ment, and personal growth. Journal of Personality, 68, 1–27. doi: 10.1111/1467-6494.00089

Hocevar, D. (1979, April). The development of the Creative Behavior Inventory. Paper presented at the annual meeting of the Rocky Mountain Psychological Association. (ERIC Document Repro-duction Service No. ED170350).

Holland, P. W., & Wainer, H. (Eds.) (1993). Differential item function-ing. Hillsdale, NJ: Lawrence Erlbaum Associates.

Horng, J. S., & Hu, M. L. (2008). The mystery in the kitchen: Culinary creativity. Creativity Research Journal, 20, 221–230. doi: 10.1080/10400410802060166

Ivcevic, Z. (2009). Creativity map: Toward the next generation of theo-ries of creativity. Psychology of Aesthetics, Creativity, and the Arts, 3, 17–21. doi: 10.1037/a0014918

Ludwig, A. M. (1992). The creative achievement scale. Creativity Research Journal, 5, 109–124. doi: 10.1080/10400419209534427

MacKinnon, D. W. (1962). The nature and nurture of creative talent. American Psychologist, 17, 484–495. doi: 10.1037/h0046541

Messick, S. (1994). The interplay of evidence and consequences in the vali-dation of performance assessments. Educational Researcher, 23, 13–23.

Messick, S. (1995a). Standards of validity and the validity of standards in performance assessment. Educational Measurement: Issues and Practice, 14, 5–8.

Messick, S. (1995b). Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50, 741–749.

Policastro, E., & Gardner, H. (1999). From case studies to robust gener-alizations: An approach to the study of creativity, In R. J. Sternberg

Dow

nloa

ded

by [

Nor

thea

ster

n U

nive

rsity

] at

18:

54 1

1 N

ovem

ber

2014