THE EVALUATIVE.VALUATIVE. AND

DIVERGENT THINKINGOFCHILDREN

MARK A. RUNCO

The Evaluative, Valuative, and DivergentThinking of Children*

INTRODUCTION The creative ideational process involves three components:problem discovery, divergent thinking, the evaluation of ideas. The

last component in this process - which is valuative and appreciative as well asevaluative and critical - has received very little attention. The present investigation wasdesigned to determine whether or not the valuative abilities of children can be reliablymeasured. This is an important test of componential theory, and of additionalimportance given the possibility that children's apparent creativity is actually uninten­tional. The view that children are only unintentionally creative assumes that theircreative behavior reflects a lack of information or skill, and this would presumably leadto inaccurate evaluations of ideas - a possibility tested herein. Another objective of thisinvestigation was to determine the degree of overlap between divergent thinking andevaluative ability. There may be a functional connection, with better divergent thinkershaving more practice evaluating ideas. In this investigation, 107 school childrenreceived three tests ofevaluative abilityand three divergent thinking tasks. Correlationalanalyses indicated that the evaluative scores were reliable (alpha =.77), and moderatelyassociated with the divergent thinking test scores (Rc =.58). The evaluative measuresalso had discriminant validity, being negatively correlated with intelligence test scores(e.g., r =. .24 with scores from the WISCR). Interestingly, the evaluative scores were,significantly influenced by instructions, with the most accurate evaluations given whenchildren were asked to estimate the number of other children who would think of eachidea (that is, estimate the popularity of ideas) rather than rate the creativity of ideas.There were no differences between boys and girls in the evaluative scores, norwas therea correlation with age.

Creativity is often viewed as a syndrome (MacKinnon, 1960, 183;Mumford & Gustafson, 1988), with a set ofdefining characteristicsand skills. Although the vital characteristics seem to vary in differentdomains of creative talent (e.g., artistic, mathematical, musical),

particular cognitive and metacognitive abilities are probably involved in mostdomains.

"This investigation was supported by the Creative Education Foundation. The author would also like to thankKelly Barclay, Theresa Ford,Joni Radio, and Luiz Vega for their assistance with the data, and Celeste Diaz forher assistance preparing the manuscript

Tentative findings from this investigation were presented at the meeting of the Western PsychologicalAssociation, in Los Angeles, CA,April. 1990.

311 Volume 25 Number 4 Foutfh Quarter 1991

The EYIIbItIYe. Vw.tlve. and DIvergentThInkIng of ChIJdren

Runco £, Okuda (1988) and Wakefield (1985, 1989), for example, isolated problemdiscovering abilities, and Hocevar (1980), Milgram £, Milgram (1976), and Runco(19800) discussed the contribution of divergent thinking. Several metacognitivecomponents have also been identified (Davidson £, Sternberg, £, Thurston, in press),including strategies for selecting certain types of information or certain informationintegration techniques.

The evaluative component of the creative process has received very little attention.This is surprising because it is a vital constituent of the creative process, and is requiredwhenever an individual selects or expresses a preference for an idea or set of ideas. Anindividual mayneed to select ideas because ofsituational or professional exigencies, orsimply because some ideas seem to be worth remembering and perhaps using. Ideaswhich have poor prospects maybe identified and then discarded. Actually, this processmay be as much valuative as evaluative, for an individual may consider theappropriateness or fit of their ideas as well as the deficiencies or undesirable aspects.Put differently, creators are probably often more appreciative than critical in theirappraisals.

Campbell (1960) and Simonton (1988) discussed evaluation as a part of the creativeprocess, and Guilford (1968) and Meeker (1969) recognized evaluation as an importantpart of the structure of intellect (SOl). In the SOl model, evaluation is one of five distinctoperations - along with divergent production, convergent production, memory, andcognition. Evaluation is not as closely tied to creativity in the SOl model as it is in thecomponential model described above.Meeker (1969) defined evaluation operations as"reaching decision or making judgements concerning criterion satisfaction (correct­ness, suitability, adequacy, desirability, etc.) of information (p. 17). In the componentialmodel of creative ideation, evaluation occurs when an individual specifically assessesthe originality or creativity of ideas.

Empirical research on evaluation includes Basadur, Wakabayashi £, Graen's (inpress) assessment of (a) active divergence, or "preference for ideation," and (b)premature convergence, or the "tendency to make premature critical evaluations ofideas." Basaduret al. reported that both attitudes were influenced by training in creativeproblem solving techniques. The trainees were members of business organizations(e.g., managers). Runco £, Vega (1990) compared parents, teachers, and other adults interms of "evaluative skill." This was defined as the accuracy of the adults at identifyingwhich ideas given by children were original. Runco £, Vega selected parents andteachers because of the possibility that the "fourth grade slump" in creativity (Torrance,1968) may be in part a reaction to pressures to conform, fit in, and follow rules - inother words, a reaction to pressures imposed by adults. Only slight differences amongthe groups were found, although experienced parents (with several children) were moreaccurate in the evaluations than inexperienced parents.

The present investigation was conducted to assess the evaluative skills of children.Because this is the first such assessment, the first important question was concernedwith the reliability of the instrument designed to assessevaluative skill. The measure ofevaluative skill used by Runco £, Vega (1990) was reliable, but their results, obtainedfrom adults, may not generalize to children. Children may lack the experience,

312

perspective, orcognitive capacity for evaluative and valuative thought In fact, in additionto testing the componential theory of creative ideation, the assessment of the reliabilityof evaluations is relevant to the question of whether or not children are intentionallycreative. Wolf & Larson (1981) suggested that the creativity of children may not reflectcreativity at all, but rather a lackofskill. In their words, the apparentcreativityofchildrencan be seen as more accidental than deliberate, i.e., children's inabilities to incorporateall of the facts and their inability to change in the light of new facts may generateresponses that are viewed as "creative" by adults by virtue ofwhat they omit rather thanwhat they include. (p. 347)

In this view, children would be expected to be poor judges of the quality of ideas.A second objective of this investigation was to determine the relationship between

evaluative! ability and divergent thinking. The componential theory described abovepredicts that there will be a moderate correlation between the two. Divergent thinkingand evaluative abilityshould not, however, be that strongly related. Bydefinition, the twocomponents are interrelated but not redundant Importantly, there may be anexperiential and functional relationship between divergent thinking and evaluative skill(Runco &Vega, 1990),with capable divergent thinkers having more practical evaluatingideas than less divergent thinkers. This practice could lead to improved evaluative skill;and presumably a moderate correlation. A third objective of this investigation was toexamine the impact of the instructions given with evaluative measures. Instructionsareoften used to enhance creative performance and to determine to what degree suchperformances are strategic (Runco, 1986b). For these reasons, one groupofchildren inthe present investigation received the evaluative measures with instructions to "rate thecreativity of each idea," and a second group received the measures with instructions to"rate the popularity of each idea." The second set of instructions was used becausepopularity is inversely related to creativity in most definitions of divergent thinking (i.e.•an original idea is an unpopular one, or one which is given by few individuals).Additionally, it is more concrete or operational than "creativity," and thus might beeasier for children to determine.

METHOD Children from a small elementary school participated in thisParticipants investigation (N =107). Fifty-seven were girls, and 50 were boys.

Forty-two were in the 4th grade, 37 in the 5th, and 28 in the 6th. Theaverage age was 10 years and 5 months (with a standard deviation of 11 months.) Theaverage WISC-R Information test standard score (available for 77 of the children) was17.6, with a standard deviation of 3.6 (compared to the standardized mean of 15.0 andsd of 3.0).

Measures and The evaluative measure used in this investigation contained threeGroups subtests. One subtest was developed from the Instances divergent

thinking test, the second from Uses, and the third from LineMeanings (all from Wallach & Kogan, 1965). More specifically, each subtest containedideas which were originally obtained by Runco & Albert (1985) by administering theInstances, Uses, and Line Meanings tests to 240 gifted and talented, and nongiftedchildren. Each of the items on the Evaluative tasks was actually an idea given by the

313

The EvaluatIve. valuative, and DMrgent ThInkIng of ChIldren

children in the investigation of Runco & Albert (1985). The children in the presentproject were, then, evaluating ideas given by children in the earlier project.

There were three pages to each of the three Evaluative measures, each representingone divergent thinking task, and each containing 25 items. The 25 itemswere randomlyselected from the ideas given by the children in Runco & Albert's (1985) project, withthe requirement that 10 of the 25 ideas were unique (given by only one child in thesample of 240).

The Evaluative measures were designed to assesshowaccuratelyexaminees identifythe creativity or popularity of a list of ideas. The actual popularity ofthe ideas on theEvaluative measures was simply the number of children who gave the idea in theinvestigation of Runco & Albert (1985). Consider the Instances question, "Name all ofthe things you can think of that are square." Of the 240 children tested by Runco &Albert (1985),30 gave "blocks" as a response, 19 gave"suitcases," and one gave"comchip." Based on these frequencies, "blocks" was a popular (and unoriginal) idea, and"com chip" was unpopular and ·original. The actual frequencies were used in thepresent investigation as the criteria for checking the accuracyof the ratings given by thechildren.

The children rated each idea on a one-to-ten scale. (The 1Olevel was used because itwas the most compatible with the popularity instructions, described below.) The ratingsgiven by each child were correlated with the frequencies found by Runco & Albert(1985), and that correlation was transformed with Fisher's r-to-z (Cohen & Cohen,1975) and used as a child's Evaluative score. Additional details about the Evaluationmeasures can be found in Runco & Vega (1990).

The children received the Evaluative measures in one oftwo Instructional conditions.One group of randomly chosen children received the Evaluation tasks with Creativityinstructions, and the other received Popularity instructions. In the former, the childrenwere asked to give high ratings (10, 9, or 8) to creative ideas, low ratings (1, 2, or 3) touncreative ideas, and moderate ratings (4, 5, 6, or 7) to moderatelycreative ideas. In thelatter, they were asked to give high ratings to popular ideas, low ratings to unpopularideas, and moderate ratings to moderatelypopular ideas. Popularitywas defined in thatthe children were asked to estimate "how manychildren in a group of 10 would thinkofeach idea?" (Hence the 1D-Ievel ratings scale.) Those receiving the Creativityinstructions were expected to give high ratings to ideas which were given by fewindividuals - that is, infrequent ideas. Note that this would result in negative Evaluativescores. Those receiving the Popularity instructions were expected to give high ratings toideas which were given by numerous children. This would lead to positive Evaluativescores.

The children in the present investigation also received the Instances, Uses, and LineMeanings tests form the Wallach & Kogan (1965) battery. Each contained threequestions. The directions were adapted from Wallach &Kogan (1965) and encouragedthe children to give as many ideas as possible.

Six subtests scores from the Kaufman Assessment Battery for Children (KABC) wereavailable for 77 of the children, as were W1SC·R Information test scores. The KABCsubtests were Hand Movements, Number Recall (both from the Sequential Processing

314

Correlations WithDivergent Thinking.

KABC. and WISC·AScores

The JoumaI Of Creative BehavIor

Scale), Gestalt Closure, Spatial Memory (both form the Simultaneous ProcessingScale), Faces and Places,and Riddles (both from the Achievement Scale).

RESULTS Coefficient alpha (an index of intemal consistency) was used toReliability assessthe reliabilityofeach measure. The alpha coefficient with the

nine Evaluative items was .77. The reliability of the divergentthinking tests was also calculated with alpha, with resulting coefficients of .77 for thethree Fluency items and .69 for the three Originality items.

Group Differences The differences between the two instructional conditions wasassessed with hierarchical multiple regression analyses. The first

regression analysis included Type of Instructions (Creative vs. Popular), Sex. and Age(in months) as predictors, and the total Evaluative score as the dependent variable.Results. indicated that the Instructional predictor was significantly related to theEvaluative score (R =.62, p < .0001). Sex and age were not significantly related to theEvaluative score (each R2 change<.006). A second regression analysiswas conductedto insure that the results of the first analysis did not reflect the order of inclusion in theregression equation. Results of this analysis indicated that age and sex were notsignificant when tested before Instructional Type (R2-changes< .07), and InstructionalType was significant when tested last (R2-change = .38, p < .001).

Transformed Scores As noted above, negative correlations were expected in theCreativity condition because a creative idea is by definition an

infrequent one, and positive correlations were expected in the Popularity condition. Forthis reason, a regression analysis was computed after the Evaluative scores weretransformed. This transformation was simply from positive or negative to absolutevalues. Itwasused to insure that the difference between the Instructional conditions wasnot onlya result ofmathematical sign (i.e.,direction of ratings), but rather was indicativeof the impact (i.e., difference from zero) of the task instructions. Results of theregression using the transformed scores indicated that the two Instructional conditionsremained significantly different (R = .40, p < .001). Sex and age were againnonsignificant (R2-changes < .004). When age and sexwere tested first, they remainednonsignificant (R2-changes< .07), and Instructional Type remained significant evenwhen tested last (R2-change < .16 P < .001).

Descriptive statistics for alI measures and the two Instructional groups are presentedin Table 1.

A canonical analysis indicated that the nine (transformed) Evalu­ative scores were marginally related to the six divergent thinkingtest scores (Rc =.58, p =.08). This relationship wasalso apparent infollow-up analyses of Instances Fluency (R =.54. P =.02) and

Instances Originality (R =.49, p =.07), using the variate of the Evaluative scores (derivedin the multivariate test) as the correlate. Similar analyses were conducted with threeKABC scores: a Sequential Processing Score (the sum of the scores from Hand

315

The EvaJuatlve. Valuative. and DMrgent ThInJdng of ChIJcIren

Movements and Number Recall), a Simultaneous Processing Score (the sum of thescores from Gestalt Closure and Spatial Memory) and an Achievement Score (the sumofthe scores from Riddles and Faces and Places). Results indicated that the Evaluativescores were unrelated to the KABC scores. There was a significant and negativecorrelation between an Evaluative composite and scores from the W1Sc.RInformationtest (r =. .24, P < .05).

TABlE 1 Descriptive Statistics for Each Group of Children InstructionalCondition

Popularity CreativityMean SD Mean SD

(n =53) (n =54)Divergent Thinking

Fluency 104.7 39.0 99.2 35.9Originality 11.8 7.9 12.4 8.6

Evaluation ScoresTotal ·1.619 1.05 0.004 1.03

DISCUSSION The most important findings of this investigation were (a) that theevaluative abilities of school children can be reliably assessed,and

(b) that there was a significant relationship between divergent thinking and evaluativeskill. A moderate correlation was found in the multivariate analyses using all of thedivergent thinking test scores and all of the evaluative scores, and a highly significantrelationship was found in the analyses specifically with the Fluency and Originalityscores from the Instances test The multivariate result implies that the associationbetween evaluative ability and divergent thinking has some generality. That significantrelationships were not found in the univariate analyses of the other measures (Uses andUne Meanings) should not come as a surprise. This idiosyncratic relationship isconsistent with the research showing clear differences among various tests ofdivergentthinking. Differences between verbal and figural tests are very common (Richardson,1986), and it is not uncommon to find differences even among verbal tests (e.g.,Michael & Bachelor, in press; Runco & Albert, 1985). The moderate correlation isconsistent with componential theory (Runco & Okuda, 1988; Runco & Vega, 1990)andsupports the prediction that divergent thinking and evaluativeability are not independentnor entirely interdependent

316

The EvaluatiYe. Valuative, and Divergent ThInking of ChIldren

Importantly, the children who evaluated the popularity of ideas were much moreaccurate than those who rated the creativity of ideas. In fact, the mean evaluative scorefor the children who received the creativity instructions was near zero (seeTable 1).Thevariability for that group was approximately equal to that of the other group of children(sds - 1.03 vs. 1.05); and many of the children receiving the creative instructions hadpositive evaluative scores. A positive score indicates that high creativity ratings weregiven to highly popular ideas (again, ideas given by many children), and low ratings tounpopular ideas. Children with the positive scores (or scores near zero) were thereforenot recognizing that creative ideas are infrequent This confirms that the implicittheories and evaluative abilities of children differ from those of adults (Runco, 1990;Runco & Vega, 1990).

The difference between the two instructional conditions may reflect the fact that thepopularity instructions were fairly specific (i.e., "give a rating to indicate how manychildren in a group of 10 would thinkofeach idea"). Additionally, developmental theorysuggests that the children are well aware of the conventionality - and thus popularity­of certain ideas, situations, and behaviors (Kohlberg, 1966). Of course, creative ideasare worthwhile as well as infrequent, and children may be able to have trouble judgingthe creativityof ideas because both need to be taken into account The data presented inthis investigation indicate that children can judge the popularity of ideas, and they mayalso know when ideas are appropriate. But perhaps because of centration tendencies,they may not be able to judge' both popularity and appropriateness at the same time.Further research with instructional manipulations could test this possibility. For now,the results given in Table 1 can be taken as indicating that there are individualdifferences in evaluative ability. .

The findings from the present investigation are inconsistent with the suggestion ofWolf & Larson (1981) that the creativity of children is more accidental than intentional.Although that hypothesis was not directly tested, the present findings indicate thatchildren are capable of recognizing infrequent, uncommon, and unpopular ideas. Thetheory that children are only accidentally creative assumes that they are lackingpertinent skills or information; but the present findings suggest that theyare capable ofcertain evaluations. Children also have other relevant skills, including problemdiscovery skill (Okuda, Runco & Berger, 1990; Wakefield, 1985), divergent thinkingability (Runco, 19800), and most importantly, strategic skills (Davidson & Sternberg,1983; Runco, 1986b). Contrary to Wolf & Larson's (1981) theory, then, the creativity ofchildren is probably not inadvertent.

Although the present data indicate that children have the potential for accurateevaluations, there is no guarantee that children will in fact use these skills. Children maynot always use their evaluative or valuative skills, just as they do not always use theirstrategic abilities. For this reason, the predictive validity of evaluative scores should beexamined in future research. It would be very helpful to know if evaluative skill predictsactual creative activity and accomplishment. The most important prediction would useevaluative skill, divergent thinking, and problem finding ability, for all three aretheoretically necessary for creative ideation (Runco, 1990).

317

Runco (19800) suggested that explicit instructions have an impact on divergentthinking primarily because they direct examinees to appropriate strategies. Thepopularity instructions used in the present investigation were fainy explicit, and mayhave given the children a specific strategy for accurate evaluations (i.e., "how manychildren in a group of 10 would give each idea?") The practical implication is thatevaluative skill might be enhanced with instructions. Harrington (1975) and Runco(l986b) demonstrated how explicit instructions can be used to increase the originalityof divergent thinking, and similar explicit instructions might be used to enhanceevaluations.

The difference between intrapersonal evaluations and interpersonal valuationsshould also be noted (Simonton, 1988; Runco £,. Vega, 1990). The children in thepresent investigation were evaluating the ideas given by other children, and this mightbe more difficult than evaluating one's own ideas. Individuals are probably more highlymotivated or interested when evaluating their own ideas,and they would be awareof theassociative history, or rationale, of the ideas. On the other hand, intrapersonalevaluations are probably highly subjective. Research in progress is comparinginterpersonal and intrapersonal evaluation skills, with the expectation that bothintrapersonal evaluations will be useful in the prediction of creative accomplishment

REfERENCES BASADUR, M., WAKABAYASHI, M.s GRAEN, G. (in press). Individual problemsolving styles and attitudes toward divergent thinking before and after training.

Creativity Research Journal.CAMPBELL, D. Blind variation and selective retention in creative thought as in other knowledge processes.

Psychological Review, 1960, 67. 380400.COHEN, J. & COHEN, P. Applied Multiple Regression/ Correlation Analysis for the Behavioral Sciences.

Hillsdale, NJ: Lawrence ErIbaum Associates, 1975.

DAVIDSON. J. E. &STERNBERG. R. J. The role of insight in intellectual giftedness. GiftedChild Quarterly.1984. 28. 58-64.

GUILFORD, J. P. Creativity, intelligence, and their educational implications. San Diego, CA: RobertKnapp/EDITS, 1968.

HOCEVAR, D.lntelligence, divergent thinking, and creativity. Intelligence. 1980,4.2540.

KOHLBERG, L Moral education in the schools: A developmental view. School Review. 1966, 75.1·30.

MACKINNON, D. The highly effective individual. In Albert, R. S. (ed.), Genius and eminence: A socialpsychologyofcreativityand exceptionalachievement Oxford: Pergamon, 1983, 114·127 (original workpublished 1960).

MEEKER, M. The structure ofintellect:Its interpretationand uses. Columbus, OH: Charles E.Merrill, 1969.

MICHAEL. W. B.& BACHELOR, P.(in press). Higher order structure -of-lntellect creativity factors in divergentproduction tests: A reanalysis of a Guilford data base. CreativityResearch Journal.

MILGRAM, R. M.& MILGRAM, N. Creative thinking and creative performance in Israeli students. Joumal ofEducationalPsychology, 1976, 68, 255-259.

MUMFORD, M. D. & GUSTAFSON, S. B. Creativity syndrome: Integration, application, and innovation.Psychological Bulletin, 1988, 103.2743.

OKUDA, S. M., RUNCO,M.A &BERGER,D. E. Creativityand thefinding andsoMng ofrealworldproblems.Manuscript under review, 1990.

RICHARDSON,A.G. Two factors of creativity. Perceptual and Motor Skills. 1986, 63, 379·384.

RUNCO,M.A.Divergent thinking and creative performance in gifted and nongifted children. EducationalandPsychological/>1easurement, 19800, 46, 375-384 .

. RUNCO, M. A. Maximal performance on divergent thinking tests by gifted, talented, and nongifted children.Psychology in the Schools. 1986b, 23, 3Q8.315.

318

The JoumaI Of CI'eatfIIe BehlMor

RUNCO, M. A Implicit theoriesand ideationalcreativity. In Runco,M. A £, Albert,R.S. (eds.), Theories ofcreativity. NewburyPark:SagePublications, 1990,234·252.

RUNCO, M. A £, ALBERT,R.S.The reliabilityandvalidityof ideational originalityin the dillergentthinking ofacademicallygifted and nongifted children. Educational and Psychological Measurement, 1985, 45,383-501.

RUNCO, M. A £, OKUDA.S.M. Problem-discovery. divergentthinking, and the creativeprocess. Journal ofYouth and Adolescence, 1988, 17,:n 1·220.

RUNCO. to\. A, OKUDA,S.to\. s THURSTON, B.J. Environmentalcuesanddivergentthinking. In Runco, M.A (ed.), Divergent thinking. Norwood,NJ: Ablex Publishing Corporation.

RUNCO, M. A £, VEGA,L Evaluating the creativityof children's ideas. Journal of Social Behavioral andPersonality, 1990,5. 439-352.

SIMONTON,D. K. &ientific genius: A psychology of science. Cambridge, MA: Harvard UniversityPress,1988.

TORRANCE, E.P.Alongitudinalexaminationof the fourth gradeslump in creativity. GiftedChJ1d Quarterly.1968,12,195-199.

WAllACH, M. A s KOGAN,N.Modes of thinking in young children. NYC:Holt,Rinehard s Winston,1965.WAKEAELD, J. F. Towardcreativity: Problemfinding in a divergentthinking exercise. Child StudyJournal,

1985, 15,265-270.WAKEAELD, J. F. Creativityand cognition: Implications for arts education. Creativity Research Journal,

)989, 2, 51-63.WOLF, F. to\. £, LARSON, G. L On why adolescent formal operators may not be creative thinkers.

Adolescence, 1981, 62, 345-348.

MarkA Runco,EC 105,CaliforniaState University, Fullerton.CA 92634.

319