Psychological Assessment2001. Vol. 13. No. 4. 423-432

Copyright 2001 by the American Psychological Association, Inc.1040-3590/01/55.00 DOI: 10.1037//1040-3590.13.4.423

Advancing the Science of Psychological Assessment:The Rorschach Inkblot Method as Exemplar

Irving B. WeinerUniversity of South Florida

This article comments on a series of 5 articles, concerning the utility of the Rorschach Inkblot Method(RIM). Two of the articles provide extensive empirical evidence that the RIM has been standardized,normed, made reliable, and validated in ways that exemplify sound scientific principles for developingan assessment instrument. A 3rd article reports a meta-analysis, indicating that the RIM and theMinnesota Multiphasic Personality Inventory have almost identical validity effect sizes, both largeenough to warrant confidence in using these measures. The other 2 articles adduce sketchy data andincomplete literature reviews as a basis for questioning the psychometric soundness of Rorschachassessment. Unwarranted skepticism should not be given credence as an adequate platform from whichto challenge abundant evidence that the RIM works very well for its intended purposes.

Because psychology is a behavioral science and assessment is afield of psychology, the credentials of psychological assessmentinstruments should be measured against scientific principles foradvancing knowledge. Data sets used to evaluate the clinical utilityof the Rorschach Inkblot Method (RIM) should accordingly bethorough and current, not incomplete or obsolete; they should beaccurate and relevant to the issue at hand, not misleading, takenout of context, or beside the point; and they should be derived fromadequately designed research studies. The information provided bythese data sets should address the following six questions concern-ing the psychometric adequacy of an assessment instrument: (a)Does the RIM comprise standardized procedures for administra-tion and scoring? (b) Can trained examiners agree reasonably wellin their scoring of the REM's variables? (c) Do reliability estimatesindicate that the RIM's scores provide reasonably accurate infor-mation by virtue of closely approximating actual or true scores?(d) Are there adequate normative reference data concerning thedescriptive statistics of the RIM's variables among various popu-lations? (e) Does the RIM demonstrate corollaries that identifypurposes for which it is reasonably valid? and (f) Can the RIMshow incremental validity when applied in decision-makingsituations?

This article addresses these six questions with respect to Ror-schach assessment, comments on the five articles previously pub-lished in this Special Series on Rorschach utility, and reaches thefollowing conclusion: By virtue of the manner in which the RIMhas been conceptualized, standardized, normed, and validated overthe past 20 years, the available data identify this inkblot method asan exemplar of sound science in the development and applicationof a psychological assessment instrument.1

I thank Mark J. Hikenroth for his helpful suggestions concerning thepreparation of this article.

Correspondence concerning this article should be addressed to Irving B.Weiner, 31716 Halliford Drive, Tampa, Florida 33624. Bectronic mailmay be sent to iweiner@hsc.usf.edu.

Standardization

The 50 years following Rorschach's (1921/1942) original pub-lication of the monograph that describes his test gave rise tonumerous disparate schools of Rorschach methodology and count-less individual variations in Rorschach technique. Despite a wealthof creative thinking and extensive research efforts, the unsystem-atic nature of these early endeavors prevented Rorschach assess-ment from establishing much claim to scientific status. Since theearly 1970s, however, with the advent of the Rorschach Compre-hensive System (Exner, 1991, 1993; Exner & Weiner, 1995),Rorschach assessment has evolved into a carefully standardizedprocedure. Exner (1993, chapter 3) prescribed an invariant set ofinstructions to be used in all Rorschach administrations, strictadherence to which ensures that Rorschach data are collected inthe same manner from all respondents and can be cumulated forresearch purposes. Exner (1993, chapters 4-10) also formulateddetailed guidelines for coding individual Rorschach responses andcombining these codes into various percentages, ratios, and in-dices. Standardization of an instrument resides in its being com-posed of specific stimuli, methods of administration, instructionsto respondents, and coding criteria to be used without exception,and differences in these components compromise the generaliz-ability of findings from one person and one situation to the next.Strict adherence to Exner's prescribed Comprehensive Systemensures the consistent comparability of data obtained from it underany circumstances.

Over the years, the Comprehensive System has become the mostwidely used approach to Rorschach assessment. The vast majorityof students who are learning the Rorschach in graduate programsare being taught the Comprehensive System: It is by far themethod most commonly used in published research studies, andabroad it is the prevalent approach in almost every country in

1 The final version of this manuscript was submitted on October 6, 1999.For this reason, with the exception of some in press material that was madeavailable, it does not include reference to any publications appearingsubsequent to that data.

423

424 WEINER

which the Rorschach is used (see Hilsenroth, 2000; Weiner,1998b). As the only carefully standardized approach to collectingand codifying Rorschach data, the Comprehensive System is theprimary Rorschach method that should be considered in evaluatingthe scientific status of the instrument. Idiosyncratic and individu-ally tailored ways of administering and coding the Rorschach willinevitably lead as in the past to inconsistent research findings.Inconsistent research findings attributable to nonstandard admin-istration and coding have often provided grist for the mill ofRorschach critics, but in fact they have no bearing at all on thepsychometric adequacy of Rorschach assessment. In a similarvein, misguided and unwarranted interpretations of Rorschachresponses can lead to errors in clinical practice and generateanecdotes that depict Rorschach assessment as a dangerous pro-cedure. Like failure to follow standard procedures, however, ill-informed interpretations reflect clinician incompetence and con-stitute an abuse of the instrument; they say nothing about thesoundness and value of the RIM when it is used properly.

It should nevertheless be acknowledged that the proceduralstandards intrinsic to the Comprehensive System do not encom-pass the interpretation of the Rorschach. Exner (1991, chapters5-10) provided some detailed guidelines for Rorschach interpre-tation, but these interpretive strategies constitute a recommendedway of working with the obtained data and are not an essentialfeature of the Comprehensive System approach. Other contempo-rary Rorschachists have endorsed Comprehensive System methodsof administration and coding while elaborating interpretive strat-egies that diverge from or extend beyond those proposed by Exner(see, e.g., Gacono & Meloy, 1994; Meloy, Acklin, Gacono, Mur-ray, & Peterson, 1997; Smith, 1994; Weiner, 1998a). The RIMresembles other widely used tests in this respect, with standard-ization residing in administration and in scoring procedures andvarious different interpretive approaches being used to determinewhat the obtained data signify. Thus, responses to the WechslerIntelligence Scales are used by some psychologists solely as indi-cations of intellectual functioning, whereas other psychologistsdraw inferences about personality characteristics as well from howrespondents behave and express themselves while taking the test.In interpreting the second version of the Minnesota MultiphasicPersonality Inventory (MMPI-2), some clinicians focus mainly onindividual scale scores and other clinicians focus on code types,and differences of opinion persist concerning how much attentionshould be given to the subtle items in the test. Such individualvariations and personal preferences in ways of approaching theinterpretive process do not detract from the basic standardizationof these instruments with respect to their administration and cod-ing, not for the RIM any more than for the Wechsler or the MMPI.

These observations on Rorschach standardization have threeimportant implications for the basic nature and utility of the RIM.First, Rorschach data are not theory bound and can be interpretedwithin the framework of diverse ways of conceptualizing person-ality processes: Provided that one believes in the existence ofpersonality states and traits, the implications of Rorschach re-sponses can be couched in whatever theoretical language oneprefers. Second, the Rorschach stimuli and instructions are essen-tially cultural free, and the instrument can be administered in astandard manner independent of a respondent's age, gender, eth-nicity, nationality, or other demographic characteristics; cross-cultural and cross-national linguistic differences may influence the

meanings that attach to the verbal content of responses, but notranslation of the test stimuli is required to obtain a valid protocolfrom anyone, anywhere in the world. Accordingly, quantitativeand qualitative research is necessary to determine ethnic differ-ences in obtained scores and their interpretive significance, but nomodification in basic Rorschach procedures is required to obtainmeaningful Rorschach data from any group of respondents, exceptoccasionally for young children.

Third, Rorschach data lend themselves both to nomothetic in-terpretation based on comparisons with available normative dataand to idiographic interpretation that allows for individualized caseconceptualization. The previously published articles by Striekerand Gold (1999) and by Viglione (1999) elaborate each of theseadvantages of Rorschach assessment; the meta-analysis by Hiller,Rosenthal, Bomstein, Berry, and Brunell-Neuleib (1999) does notaddress these points; and the articles by Hunsley and Bailey (1999)and by Dawes (1999) are silent with respect to them.

Intercoder Agreement

Assessment psychologists generally concur that test scores canserve useful purposes only if examiners can agree on what thesescores should be. Without substantial agreement concerning thescores that should be assigned to test variables, these variableshold little promise for demonstrating internal consistency, tempo-ral stability, or consistent extratest corollaries. Opinions vary,however, concerning the most appropriate way of evaluating in-tercoder agreement, especially in the case of assessment instru-ments that involve multiple complex coding categories. McDowelland Acklin (1996), Meyer (1997a, 1997c), and Wood, Nezworski,and Stejskal (1996, 1997) among others have engaged in livelydebate, concerning such matters as (a) the relative appropriatenessof calculating percentage agreement versus Kappa coefficients indetermining intercoder agreement for Rorschach responses and (b)the necessity of using entire responses versus response segments asthe unit on which to base these calculations. By far the mostextensive and methodologically sophisticated work on this topichas been conducted by Meyer (1997a), who reported a meta-analytic study of interrater reliability data published in Rorschachresearch articles, using Comprehensive System variables that ap-peared in the Journal of Personality Assessment from the begin-ning of 1992 to the end of 1995. Using Kappa to assess these datafor the chance-corrected interrater reliability of 10 response seg-ments (e.g., Location, Determinants, Form Quality, etc.), Meyerobtained coefficients for individual segments, ranging from 0.72to 0.98, with a mean value of 0.88. As Meyer noted, Kappa valuesgreater than 0.75 are generally considered to demonstrate excellentbeyond-chance agreement.

The present status of knowledge that concerns whether theRorschach can be coded reliably is addressed in two of the fiveseries articles under discussion here, those by Viglione (1999) andby Hunsley and Bailey (1999). In light of the previously statedcriteria for a sound scientific approach to advancing knowledge,the comments of these authors should be introduced with someattention to the data sets on which they relied in drawing conclu-sions about the psychometric adequacy of the Rorschach method.As noted by Meyer (1999b) in his introduction to this SpecialSeries, the authors of each contribution were provided with ab-stracts of 445 Rorschach articles identified by PsycLIT as having

SPECIAL SECTION: FOUNDATIONS OF THE RORSCHACH 425

been published during 1977-1997 in five journals: Assessment,Journal of Clinical Psychology, Journal of Consulting and Clini-cal Psychology, Journal of Personality Assessment, and Psycho-logical Assessment. Viglione (1999) indicated that he siftedthrough these and some additional articles appearing elsewhere;that he put aside reviews of research, reports of surveys, casestudies, theoretical and methodological discussions, and other sec-ondary or nonempirical contributions; and that he focused hisattention on original research studies. His reference list contains195 items, of which 139 are asterisked to identify them as empir-ical studies that he evaluated in preparing his article. Hunsley andBailey (1999) did not identify the procedures they followed inexamining the relevant literature. Their reference list comprises128 items, of which just 28 appear to be original research reports.The remaining 102 items consist of commentaries, surveys, andsecondary sources, that is, reviews of research published by others.

Turning to the substantive matter at issue, Viglione (1999) cameto the following conclusion concerning intercoder agreement onthe Rorschach:

In preparing this article, I had the opportunity to review hundreds ofinterscorer reliability practices and coefficients for all types of scoreswith all sorts of base rates; with dichotomous, multiple categorical, orrating-scale distributions; completed in Asia, North America, andEurope; and with all sorts of reliability statistics. The fact is that justabout everybody reports adequate interscorer agreement, (p. 252,footnote 4)

Hunsley and Bailey (1999) did not seriously question whetherthe Rorschach can be coded reliably, nor did they review anyevidence that suggested that it cannot. They limited their discus-sion on this matter to raising two points for consideration. First,they argued that some resolution must be reached, concerning themost appropriate methodology for assessing intercoder agreementon the Rorschach. They may be correct that some one methodworks best for this purpose and should be the one always used,although varied approaches to most problems in psychology haveusually proved scientifically enriching rather than disadvanta-geous. More to the point, however, it appears not to matter at allwhich methods are used in assessing Rorschach scoring reliability.As Viglione (1999) has pointed out, adequate intercoder agreementhas been demonstrated virtually wherever, whenever, and bywhomever such demonstration has been attempted and with what-ever reasonable methods have been used.

Second, Hunsley and Bailey (1999) expressed concern thatreliable Rorschach coding depends on the coding skills of individ-ual examiners and that there remains "the significant question ofhow reliably the Rorschach is scored in routine clinical practice(i.e., field reliability)" (p. 268). Indeed, the adequacy with whichclinical practitioners use methods of evaluation and treatment mustalways be of concern among helping professionals. However, poorpractices reflect on the competence of the clinicians responsiblefor them, not necessarily on the soundness of the methods they areattempting to use. As a case in point, I do not recall ever havingseen a field study of the reliability with which Wechsler AdultIntelligence Scale (WAIS)/WAIS-R/WAIS-III examiners as-signed 0, 1, or 2 points to responses on the Comprehension,Similarities, and Vocabulary subtests. However, the Wechslermanuals provide explicit criteria and scoring examples for 0-, 1-,and 2-point answers, and conscientious examiners who study the

manual and receive some training should be able to achieve goodagreement in this scoring. On this basis, it is generally assumedthat clinicians in practice are scoring the Wechsler tests reliably. Ifthey are not, the shame is theirs, and the WAIS/WAIS-R/WAIS-III is not held accountable for their incompetence.

So let it be with the Rorschach. The Comprehensive Systemworkbook (Exner, 1995) provides detailed coding criteria, numer-ous examples, and an extensive set of practice exercises; and theabundant research noted by Meyer (1997a) and Viglione (1999)demonstrates that examiners can readily be trained to achieveadequate intercoder agreement. Whatever grievous errors individ-ual practitioners may commit in coding Rorschach protocols willspeak to their lack of diligence and give no fault to the intrinsicpsychometric adequacy of the instrument and its coding criteria.These considerations do not obviate continued effort to improvethe specificity of coding criteria as a means of promoting inter-coder agreement, without which any instrument, including theWechsler scales as well as the Rorschach, will have limited valid-ity and utility.

Finally with respect to intercoder agreement, those who remainunconvinced of the reliability of the Rorschach in this respect,even while failing to muster any data that suggest that suchreliability is elusive, are paying insufficient attention to the impli-cations in this regard of test-retest findings with the RIM. Asdiscussed next in considering the retest reliability of the instru-ment, the vast majority of variables coded in the ComprehensiveSystem have consistently demonstrated substantial stability coef-ficients over intervals, ranging from 7 days to 3 years. As McCann(1998) and Meyer (1997c) have pointed out, coding accuracy isalways a nested component of a test-retest design. Only if both thefirst and second set of scores in a retest study have been assignedreliably can these scores correlate highly with each other. Hence,the substantial stability coefficients repeatedly demonstrated fornumerous Rorschach variables bear irrefutable witness to goodinterrater reliability among the many persons who participated incoding these records.

Reliability

The reliability of Rorschach data coded, according to the Com-prehensive System, has, as just noted, been demonstrated in aseries of retest studies that are summarized by Exner and Weiner(1995, pp. 21-27). Almost all of the variables coded in the systemthat relate to trait characteristics of individuals have shown sub-stantial short-term and long-term stability in adults. Most of thesevariables demonstrate retest correlations above 0.75, and some ofthese correlations (e.g., the Affective Ratio and the EgocentricityIndex) approach 0.90. The only variables in the system that showlow stability coefficients among adults are inanimate movement(m) and diffuse shading (Y), both of which are conceptualized asindexes of situational distress and are expected to prove unstableover time. Among children, 3-week retest studies identify stabilitycoefficients similar to those found in adults. However, as would bepredicted from the evolving nature of personality during the de-velopmental years, nonpatient young people do not begin to showadult levels of Rorschach stability over a 2-year period until theyreach age 14 (Exner, Thomas, & Mason, 1985).

Consistent with this evidence that the RIM is a scientificallysound instrument with respect to its reliability, Viglione (1999)

426 WEINER

concluded from his literature review that "the great majority ofRorschach comprehensive system (CS) variables and configura-tions have shown impressive consistency reliability" (p. 252).Hunsley and Bailey (1999) similarly stated that "it does appear thatmany of the scales central to the Comprehensive System can haveadequate reliability.... Evidence ... generally supports the reli-ability (often test-retest reliability) of scales commonly used inresearch that are scored by trained raters" (p. 268). With respect toHunsley and Bailey's reference to scoring by trained raters, andwith further relevance to the previous discussion of field reliabil-ity, one would hope that neither researchers nor clinicians wouldattempt any coding of Rorschach responses without having beentrained to do so.

Normative Reference Data

None of the five articles being discussed addresses the avail-ability of normative standards for Rorschach variables, eventhough adequate norming is identified as a requisite in test con-struction and evaluation in the Standards for Educational andPsychological Testing (American Psychological Association[APA], 1985). With respect to the psychometric adequacy of theRIM in this regard, the development of the Comprehensive Systemincluded the compilation of descriptive statistics for each of itscodes and summary scores on a sample of 700 nonpatient adults,ages 18 to 70. As described by Exner (1993, chapter 12), thissample was randomly selected from a larger group of 1,332 vol-unteer participants and was stratified to include an equal number ofmales and females and 140 respondents from each of five geo-graphic areas across the United States (Northeast, South, Midwest,Southwest, and West). The normative reference sample is alsogenerally representative of United States demographic patterns ofmarital status, level of education, socioeconomic status, andurban-suburban-rural residence, and it includes 19% of AfricanAmerican, Hispanic, and Asian American respondents. Referencedata are also available for 1,390 nonpatient young people, ages 5to 16, and for groups of adult psychiatric patients, including 320schizophrenic inpatients, 315 depressed inpatients, 440 psychiatricoutpatients, and 180 patients with diagnosed character disorder.

The compilation of these reference data has helped to establishthe RIM as a standardized assessment instrument and has aidedassessment clinicians considerably in their efforts to identify per-sonality assets and liabilities in persons they examine. However,normative standards remain an unfinished aspect of the Compre-hensive System. Because of the manner in which Exner's nonpa-tient volunteer respondents were recruited, they may constitute arelatively well-adjusted rather than a generally representative sam-ple of United States adults. With respect to utilization of theComprehensive System abroad, moreover, some adjustments incoding criteria and some modifications in normative expectationsmay be required in order to apply the Comprehensive Systemeffectively in certain countries or with certain cultural groups. Onthe other hand, there is good reason to believe that general prin-ciples of Rorschach interpretation are universally applicable andonly slightly affected by cross-cultural differences (Butcher,Nezami, & Exner, 1998; Weiner, 1998a, chapter 2; 1999). Prelim-inary results from a multinational collaborative normative studyinvolving 2,250 respondents examined in 12 different countrieshave, in fact, identified remarkably similar ranges of values for

many Comprehensive System variables (Erdberg & Shaffer,1999).

Discussions of test development have tended in general tooverlook the importance of normative data not only for enhancingthe clinical utility of assessment instruments but also for demon-strating aspects of their construct validity. Demographic charac-teristics, such as a person's age or gender, can be particularlyhelpful in this regard, because they constitute observed variablesthat have little, if any, error variance and are completely indepen-dent of test findings. Should demographic differences on person-ality test variables emerge precisely as would be predicted bywell-documented aspects of personality theory, normative data canprovide powerful construct validation. The Comprehensive Sys-tem normative data for young people, ages 5 to 16, do in factprovide such validation by mirroring closely certain expectedmaturational changes, during childhood and adolescence.

Weiner (1996) has previously called attention to three agedifferences of this kind. First, consistent with the ordinary ten-dency of personality characteristics to become increasingly stableduring the developmental years, longitudinal findings indicatesteadily increasing retest correlations for Rorschach variables asyoung people age from 8 to 16 years. Second, the normal matu-rational tendency for children over time to become emotionallyless intense and more reserved is reflected in a steadily decreasingratio from age 6 to 16 of the number of color-form (CF) responses,which are considered a corollary of relatively unmodulated pat-terns of emotionality, to the number of form-color (FC) responses,which are considered a corollary of relatively modulated patternsof emotionality. Third, children are typically observed to becomeless self-centered as they mature and are increasingly capable offocusing their attention on other people as well as themselves, andthe Egocentricity Index, which is considered to signify the extentto which a person is self-focused, diminishes in a steady linearfashion from age 5 (M = 0.69) to age 16 (M = 0.43). In a similardata analysis based on the Comprehensive System age-groupnorms, Wenar and Curtis (1991) demonstrated longitudinal Ror-schach changes consistent with predictions from developmentalpsychological data, concerning increases over time in cognitivecomplexity, precision of thinking, and conformity to socially ac-ceptable ways of thinking.

Validity

In the final analysis, even the most carefully standardized,highly reliable, and thoroughly normed assessment instrumentserves little purpose unless it can be demonstrated to have mean-ingful correlates. The nature and extent of an instrument's corol-laries define its validity and utility, and the five articles underdiscussion are addressed mainly to this feature of the RIM. Threeof the articles ( Strieker & Gold, 1999; Killer et al., 1999; Vigli-one, 1999) address methodological considerations as well as sub-stantive findings in assessment validity research, and a brief re-view of three such considerations can help to introduce commentson the substantive findings.

Methodological Considerations

Assessment and measurement psychologists generally concurwith respect to three considerations in designing and evaluating

SPECIAL SECTION: FOUNDATIONS OF THE RORSCHACH 427

empirical studies of test validity. First, the validity of assessmentinstruments that are multidimensional in nature cannot be capturedby a single numerical value or narrative statement. Each of thescale scores generated by such instruments has its own validitycoefficients, and each scale's coefficients vary with the purposesfor which it is used. Depending on the correlates of their individualscales, most instruments demonstrate greater validity for somepurposes than for other purposes; depending on their nature, someinstruments demonstrate greater validity for certain purposes thanother instruments. The RIM is by nature a measure of personalityprocesses, and Rorschach assessment should accordingly be ex-pected to measure variables that are determined mainly by person-ality characteristics; it should not be expected to produce signifi-cant correlations with phenomena or events that are attributablelargely to nonpersonality variance, nor should its validity be im-pugned when it fails to do so.

Second, the validity of assessment instruments should be judgedprimarily from their correlations with observed rather than inferredvariables. Observed variables comprise directly noted features ofhow people think, feel, and act; inferred variables consist ofhypotheses about how people are likely to think, feel, and act thatare derived from indirect sources of information. Because person-ality assessment instruments are inferential measures, their corre-lations with each other provide little information that concernstheir validity for explaining or predicting observed behavior; it ispossible for a set of inferential measures to correlate substantiallywith each other while showing no consistent relationship to ob-servable behavior. Viglione (1999) elaborated this consideration inhis article and stressed the importance of validating Rorschachvariables against "behavioral, real-life criteria," as did Hilsenroth,Fowler, Padawer, and Handler (1997) in research that demon-strated how well the RIM can identify accurately pathologicalexpressions of narcissism.

Correlations between inferential measures are especially limitedin their significance for validity when the measures involve dif-ferent methods of approach. The RIM, for example, is a largelyindirect assessment method, involving a relatively unstructured setof test materials and instructions to which people respond withlittle conscious awareness of what their answers might signify. TheMMPI and similar self-report inventories, by contrast, are largelydirect assessment methods, involving relatively structured materi-als and instructions; by virtue of how their clinical scales weredeveloped, these inventories may include subtle as well as obviousitems, but people respond to them for the most part with consid-erable awareness of what their answers might signify. An exten-sive conceptual and empirical literature documents that methodvariance can produce substantial differences between Rorschachand MMPI findings, even on variables that are assumed to measuresimilar constructs, without invalidating either instrument as a wayof measuring certain aspects of these constructs (see Bomstein,1999; Ganellen, 1996a; McClelland, Koestner, & Weinberger,1989; Masling, 1997; Meyer, 1997b). Strieker and Gold (1999)concluded a thorough discussion of this literature by noting that "itis not good science to study the validity of the Rorschach inconvergent validity studies that use self-report instruments astarget criteria" (p. 242).

From a clinical perspective, judging the validity of Rorschachscales from their correlations with self-report scales has someadditional shortcomings. Because these measures differ in the

degree to which respondents are consciously aware of what theiranswers might signify, they differ in their susceptibility to impres-sion management. Respondents' test-taking attitudes and the im-pression they would like to give are consequently less likely toinfluence the nature and extent of what they reveal about them-selves when they are doing the RIM than when they are doing theMMPI. Meyer (1999b) has presented a detailed analysis of data,demonstrating that the degree to which Rorschach and MMPIscales correlate is moderated by similarities and differences in thestyle of approach that test respondents bring to their task. Also ofnote is the common observation that divergence between relativelystructured and relatively unstructured measures of similar phenom-ena can often be very informative, sometimes even more usefulthan convergence between the measures, and by no means inval-idates either finding (see Finn, 1996; Ganellen, 1996b; Weiner,1993). Such complementary use of tests is considered further in thefinal section of this commentary, which concerns incrementalvalidity.

Third, in addition to addressing the use of specific scales forspecific purposes and correlating these scales with observed vari-ables, validation research with Rorschach scales should focus onparticular observed variables that are conceptually linked to thesescales in ways that would predict a consistently significant rela-tionship between them. It is not sufficient merely to anticipate thatRorschach scores in general will correlate with personality-determined behavior in general. Instead, an informed conceptualapproach to assessment research formulates predictions on thebasis of identifying particular personality characteristics that arebelieved to account both for a particular test score that measures itand for a particular behavior that reflects it. A positive finding thengoes beyond demonstrating what goes with what, which consti-tutes criterion validation, and provides a heuristic explanation ofwhy a test score measures what it does, which is the essence ofconstruct validation (see Weiner, 1995). In the absence of suffi-cient conceptualization to identify criterion variables with whichtest scales should be expected to correlate, researchers run the riskof giving as much weight to irrelevant as to relevant variables.Failure to avoid irrelevance in the selection of criterion variablescan result in penalizing a test for not doing something in theabsence of any reason to believe that it should.

Substantive Findings

Turning to the substantive conclusions about validity in thesefive articles, Strieker and Gold (1999) stated in their review of theempirical literature that, with respect to basic personality structureand the dispositions of people to behave in certain ways, "... inrecent years there has been the accumulation of an impressivebody of studies that have indicated that the Rorschach and otherrelated methods reliably and validly tap into such processes" (p.241). Viglione (1999) came to the following similar conclusion inhis extensive analysis of published research:

Taking the time to read and to understand the empirical literature inrefereed journals over the last 20 years leads to the conclusion that theRorschach variables are useful for many purposes in clinical, forensic,and educational settings. This conclusion rests on a synthesis of theempirical literature emphasizing ecologically valid, behavioral, real-life criteria.... The assumption that the Rorschach is not useful. . . ismistaken and contrary to the evidence, (p. 260)

428 WEINER

A careful reading of Viglione's review and of the primarysources he cites should leave little doubt concerning the soundnessof his conclusions. Lending considerable additional support to thisconclusion is the Miller et al. (1999) article mentioned only brieflythus far, which consists of a meta-analytic study, comparing cri-terion validity evidence for the RIM and the MMPI. Miller et al.introduced their analysis by reviewing prior efforts of this kind,with particular attention to frequently cited meta-analyses by At-kinson (1986) and by Parker, Hanson, and Hunsley (1988) and toa methodological critique of these meta-analyses by Garb, Florio,and Grove (1998). The Garb et al. (1998) article was subsequentlyfollowed by a rebuttal from Parker, Hunsley, and Hanson (1999)and a reply from Garb, Florio, and Grove (1999). Hiller et al.delineated numerous methodological shortcomings in both theearlier meta-analytic reviews and in the Garb et al. (1998) reanal-ysis of the earlier data, and they described in detail the proceduresthey followed to correct these flaws and enhance the reliability oftheir findings.

Hiller et al. (1999) began their meta-analysis with 4,370 MMPIarticles and 1,793 Rorschach articles appearing in the literaturefrom 1977-1997. Using random sampling procedures and limitingtheir database to research studies in which there was at least oneexternal criterion variable and to associations between variablesthat could be expected to be significant, they eventually examinedstudies involving 5,007 MMPI protocols and 2,276 Rorschachprotocols.

The results they obtained led Hiller et al. (1999) to numerousimportant conclusions, of which the following four are especiallygermane to the issue of Rorschach validity:

1. The RIM and the MMPI have equivalent validity effect sizes.Hiller et al. (1999) found unweighted mean validity coefficientsof 0.29 for Rorschach variables and 0.30 for MMPI variables, andthey were able to demonstrate that these validity estimates for thetwo instruments do not differ significantly from each other. Hilleret al. commented as follows on the reliability of their findings:"The methodological features of this study, including randomsampling from the published literature, expert judgments for in-clusion of validity evidence, and the use of accepted effect sizeestimation techniques, lend greater credibility to these resultscompared with those from previous efforts" (p. 291).

2. The magnitude of the obtained effect sizes for both the RIMand the MMPI is substantial and warrants user confidence inapplying both instruments for their intended purposes. Hiller et al.(1999) referred to Cohen's (1988) observation that correlationcoefficients on the order of 0.30 are near-maximum outcomes inrelating personality measures to real-life criteria, and they con-cluded that "validity for these instruments [Rorschach and MMPI]is about as good as can be expected for personality tests" (p. 291).

3. The RIM and the MMPI demonstrate areas of superiority toeach other in the validity coefficients they generate. Average effectsizes found by Hiller et al. (1999) indicate that Rorschach variablesare somewhat superior to MMPI variables in predicting objectivecriterion variables, such as unambiguous behavioral outcomes(e.g., treatment attendance) and discrimination of objectively dif-ferent groups (e.g., patients with closed head injury vs. normalcontrols). The MMPI, on the other hand, shows somewhat highereffect sizes than the RIM in studies that use psychiatric diagnosisand self-report measures as criterion variables. Hiller et al. notedthat shared method variance probably accounts for the relatively

high correlations of the MMPI with self-reports and psychiatricdiagnoses, which are based largely on self-reports. Likewise, inproviding confirmation for the previously referenced conceptualand empirical literature concerning differences between these mea-sures, their data may reflect the particular sensitivity of the RIM topersistent behavioral dispositions.

4. There is little likelihood that unpublished research studiescontain findings that would alter the results and implications ofthis meta-analysis. Hiller et al. (1999) determined on the basis ofstatistical procedures that thousands of unpublished studies withnegative results would have to be hidden away in file drawers todetract substantially from the effect sizes they obtained. Althoughthey acknowledged that "some bias may be present because ofunrepresented unpublished studies," they concluded that "the like-lihood that these unpublished studies would reduce the signifi-cance of results below traditional levels is very small" (p. 290).

This brings us to the article by Hunsley and Bailey (1999), whoconcluded with respect to the validity of the RIM that "there islittle scientific evidence to support the clinical utility of the Ror-schach [and] currently no scientific basis for justifying the use ofRorschach scales in psychological assessments" (p. 266). Hunsleyand Bailey even went so far as to say that "the ComprehensiveSystem, as a whole, does not meet the requirements set out inprofessional standards of practice such as the Standards for Psy-chological and Educational Testing ([APA,] 1985)" (p. 271).

Is it possible that these authors read the same literature that ledStrieker and Gold to conclude that accumulating evidence providesan impressive demonstration of Rorschach validity, that led Vigli-one to conclude that Rorschach variables are useful for manypurposes, and that resulted in Hiller et al.'s (1999) obtaining ahighly respectable average validity coefficient for Rorschachscales, equivalent to the validity of MMPI scales? Foregoingspeculation in this regard, let it be said that Hunsley and Bailey'ssweeping indictment of Rorschach assessment as failing to meetprofessional standards of practice, ignoring as it does abundantevidence to the contrary, is without substance.

As an important case in point, let us examine the evidence citedby Hunsley and Bailey in support of their opinion that the RIMlacks validity. Hunsley and Bailey based their challenge of Ror-schach validity largely on three lines of research. First, they foundfault with the methodology of the previously mentioned meta-analytic studies by Atkinson (1986) and Parker et al. (1988) andobserved that these studies have frequently been credited withproviding support for the scientific merit of Rorschach assessment.As far as Hunsley and Bailey are concerned, "it is clear that currentassessments of the general validity of the Rorschach rely almostentirely on invalid or extremely limited meta-analytic evidence. . . . [and] this meager evidence cannot be used to substantiate theclinical utility of the test" (p. 269). The methodological adequacyof the Parker et al. meta-analysis has also been challenged by Garbet al. (1998), whose opinions as previously noted evoked first arebuttal (Parker et al., 1999) and then a reply (Garb et al., 1999).In contrast to the just quoted Hunsley and Bailey (1999) positionthat there is insufficient scientific basis to justify the use ofRorschach scales in psychological assessments, Parker et al.(1999) asserted that the Rorschach "is valid, stable, and reliableunder certain circumstances" and that "psychologists should beprepared to accept that there may be domains in which the Ror-schach has greater utility than the MMPI" (pp. 291 & 292).

SPECIAL SECTION: FOUNDATIONS OF THE RORSCHACH 429

However this may be, the extensive and methodologically so-phisticated meta-analysis of Rorschach validity findings reportedby Hiller et al. (1999) appear to render moot any disputes about theadequacy of the Atkinson and Parker et al. (1988) methodology.Moreover, the magnitude of the effect sizes found by Hiller et al.for Rorschach scales refutes any assertion that the clinical utility ofthe Rorschach lacks substantiation by adequate evidence. Hunsleyand Bailey (1999) did not have the Hiller et al. data available tothem when they prepared their article, and they cannot be faultedfor not considering its implications. Nevertheless, the fact remainsthat the Hiller et al. data weaken considerably their argument thatonly limited or suspect meta-analytic data are available to supportthe validity of Rorschach assessment. Of further note in this regardare two other recent meta-analyses that have documented thepredictive validity of specific Rorschach scales. Meyer and Han-dler (1997), examining 20 effect sizes involving 752 participants,found a population correlation of 0.44 between the RorschachPrognostic Rating Scale and independent ratings of psychologicaltreatment outcomes 1 year later. Bornstein (1999) examined 21effect sizes, involving 538 respondents and found a validity coef-ficient of 0.37 in using the Rorschach Oral Dependency scale topredict independently observed dependency-related behaviors.

Second, Hunsley and Bailey (1999) argued that unresolvedissues, concerning the effect of response frequency (R) on obtainedscale scores, undermine any potential validity that the RIM mightotherwise demonstrate. If this were indeed the case, then logicalreasoning indicates how remarkable it is that the Rorschach hasdemonstrated substantial effect sizes in the total 1977-1997 corpusof research studies, despite such a handicap; logic suggests furtherthat Rorschach validity coefficients would be even higher thanthose found by Hiller et al. if this imputed problem with R werecorrected. As for the problem itself, however, there do remainsome issues to be resolved. On the one hand, recent work byMeyer (1999a), using refined statistical modeling procedures, hasdemonstrated that R by itself does not moderate the convergentvalidity of Rorschach scales with the MMPI and that prior findingsthat suggested otherwise were the result of sampling error. On theother hand, Meyer (1998) has also reported an average correlationof .25 between R and 33 criterion scores that contribute to variousComprehensive System indices. Additionally, Weiner (1998a, p.109) has observed that short records with 14 to 19 responses maybe valid as far as they go but, as a consequence of the respondent'sguardedness, may not fully reveal the extent of either the person'scapacities or his or her concerns. Further research is needed toelucidate appropriate ways of taking unusual protocol length intoconsideration in the interpretive process.

Third, Hunsley and Bailey (1999) gave considerable weight toresearch, showing limited convergence between Rorschach andMMPI scales, from which they infer limited convergent validity onthe part of the RIM. As noted earlier, attempts to validate oneinferential personality measure against another such measure, inthe absence of observable criterion variables, and especially tomake such an attempt in the face of considerable method variancebetween them, constitutes simplistic and largely uninformativemethodology. Hunsley and Bailey acknowledged these consider-ations to some extent and commented in particular on the differ-ences in the level of conscious awareness with which respondent'sgive their answers to different kinds of tests. However, they gaveno ground in this regard, concluding instead that "claims that the

Rorschach's low-convergent validity reflects its virtues rather thanits limitations must be supported by empirical evidence, not simplyrhetoric" (p. 270).

Such empirical evidence is ready to hand in the Hiller et al.(1999) meta-analysis, which as previously mentioned identifiestypes of real-life criterion variables that are predicted better byRorschach than by MMPI scales. Should one still wish to embraceRorschach-MMPI correlations as a valuable index of validity, whynot conclude that limited convergence between these measuresreflects badly on the validity of the MMPI? The answer to thisquestion touches on the issue of the incremental validity of Ror-schach scales, which is discussed by Hunsley and Bailey, Striekerand Gold (1999), and Viglione, and which is the central focus ofthe article in this Special Series by Dawes (1999).

Incremental Validity

The incremental validity of an assessment instrument consists ofthe extent to which its scores are likely to increase the accuracy ofpredictions derived from other sources of information, includingscores on other tests. Not surprisingly, the articles under discussionarrive at quite different conclusions in this matter. Consistent withtheir blanket disbelief in the validity of the RIM, Hunsley andBailey (1999) gave short shrift to its incremental validity: "Thelimited evidence bearing on this question to date does not support,in general, the incremental validity of Rorschach scales" (p. 271).They cited two sources of information in support of this conclu-sion. First, they referred to a conclusion reached by Garb (1984)that Rorschach data do not add to the accuracy of personalityassessments based on demographic or self-report data. They con-tinued on to acknowledge that none of the studies reviewed byGarb involved use of the Comprehensive System. Given that theComprehensive System is the only adequately standardized andthe most widely used Rorschach method, how can a 1984 reviewof research exclusive of Comprehensive System studies be takenseriously in 1999 as a basis for casting aspersions on the incre-mental validity of the Rorschach? Garb (1998, p. 20) has recentlyrestated his conclusion that Rorschach assessment shows little orno incremental validity, but the publication dates of the referenceshe cites in support of his opinion range from 1954 to 1982 andhave no bearing whatsoever on the Comprehensive System.

The only other basis cited by Hunsley and Bailey for theirnegative opinion of the Rorschach's incremental validity com-prises two studies in which the diagnostic efficiency of MMPIscales was reportedly not enhanced by combining them withComprehensive System Rorschach indices (Archer & Gordon,1988; Archer & Krishnamurthy, 1997). A close look at these twostudies suggests that scholars should not be too quick to interpretthem as challenging the utility of Rorschach indices. As shown inTable 2 of the Archer and Gordon study, the traditional cutoffscore of >3 on the Rorschach Schizophrenia Index achieved a69% hit rate in differentiating schizophrenic from nonschizo-phrenic patients in their sample, whereas the traditional cutoffscore of >69 on MMPI Scale 8 (Schizophrenia [Sc]) achieved60% accuracy. When more conservative cutoff scores were exam-ined, the classification accuracy increased to 80% for the Schizo-phrenia Index > 4 and to 76% for Sc > 74. If there is anydifference here, the Rorschach scale outperformed the MMPIscale, and it would seem legitimate to ask at which doorstep the

430 WEINER

problem should be laid if using both tests together did not enhanceclassification accuracy. As for the Archer and Krishnamurthystudy, these authors found that the Rorschach Depression Indexdid not add incrementally to the MMPI in predicting a diagnosis ofdepression. On the other hand, their data indicate that two com-ponents of the Depression Index, the Vista response and theAffective Ratio, did enhance the identification of depression overand above information provided by the MMPI.

Dawes (1999) has contributed a methodological article to thisseries in which he describes and illustrates two data analysistechniques for assessing the incremental validity of a Rorschachvariable. Using the Ego Impairment Index (Ell; Perry, Viglione, &Braff, 1992) as his example, he found that it correlated signifi-cantly with severity of psychopathology, as operationalized byseveral diagnostic categories. It is noteworthy that Dawes, who hasbeen widely quoted for writing in 1994 that the Rorschach is a"shoddy" instrument that "is not a valid test of anything" (Dawes,1994, pp. 123 & 146), has demonstrated with his data analysis thatit is very good indeed for something important and as good as theMMPI. Table 1 in Dawes's article shows correlations between theEII and two types of ratings for severity of psychopathology of .35and .20, both significant at the 1% level of confidence. The tablealso shows almost identical correlations of .31 and .27 betweenthese ratings and average scale elevation on the MMPI. The MMPIGoldberg index for differentiating between psychotic and neuroticdisorder fared less well than the Rorschach EII in this analysis,with correlations of .17 and —.03 with the two ratings of severityof psychopathology.

With respect to incremental validity, however, Dawes (1999)concluded that the EII "had only slight incremental validity overand above the number of responses and form quality, and even lesswhen the average MMPI elevation and L. R. Goldberg's (1965)formula for predicting psychosis versus neurosis were enteredbefore these Rorschach variables" (p. 297). This conclusion isdifficult to reconcile with the data shown in Table 3 of the Dawesarticle. When the R2 values in this table are unsquared to indicatethe multiple Rs that obtain, the prediction of maximum diagnosticseverity from combining average clinical scale elevations and theGoldberg Index for the MMPI reaches .37. When Rorschachresponse total, form quality, and the EII are added, the R withmaximum diagnostic severity increases to .49. This increasedpredictability from an R of .37 to an R of .49 is not in my opinionan instance of "slight incremental validity." Rather, Dawes's dataappear to demonstrate substantially increased validity, with theRorschach variables adding a respectable amount of informationbeyond what could be obtained from the clinical scales of theMMPI.

On the other side of this debate, Strieker and Gold (1999) andViglione (1999) cited numerous research studies involving manydifferent Rorschach scales and diverse criterion variables to war-rant the conclusion that Rorschach assessment has considerableincremental validity and provides valuable information in person-ality evaluations beyond what is learned from interview and self-report methods. As a current example in this regard, Perry, Geyer,and Braff (1999) reported a study in which Rorschach indices ofimpaired reality testing (low X-%) and disordered thinking (ele-vated WSum6) predicted physiological correlates of schizophrenia(prepulse inhibition of the startle response) significantly and overand above predictions from scores derived from two structured

interviews, the Scale for the Assessment of Positive Symptoms andScale for the Assessment of Negative Symptoms.

Strieker and Gold (1999) and Viglione (1999) also both arguedthat the determination of whether personality assessment instru-ments are incrementally valid should go beyond nomothetic ap-proaches, which have implications for group averages and tenden-cies, and include as well idiographic approaches, which haveimplications for individual case conceptualization. To elaboratethis nomothetic-idiographic distinction in the present context, let itbe noted that Rorschach critics who have considered the instru-ment lacking in incremental validity have typically based theirconclusions on nomothetic data analysis, as illustrated by MMPI-Rorschach studies to which Hunsley and Bailey referred and by themethodology proposed by Dawes. Lack of incremental validityfrom this nomothetic perspective means that combining one testwith another, as in adding the RIM to an MMPI, or vice versa, doesnot significantly increase the percentage of a participant samplethat is correctly classified as having some characteristic orcondition.

Whatever import may attach to such a nomothetic lack ofincremental validity, it bears little relevance to the assessment ofpsychological functioning in individual patients and clients seen inclinical practice. Most assessment clinicians who use the RIM andMMPI conjointly are familiar with cases in which (a) a normalrange MMPI occurs together with Rorschach indications of sub-stantial deviations from normative personality functioning (a com-mon occurrence in custody, personnel selection, and other admin-istrative evaluations in which respondents are trying to make afavorable impression); or (b) a valid but guarded Rorschach pro-tocol, providing minimal information occurs together with a validand highly informative MMPI (a common occurrence when re-spondents are disinclined to reveal very much about themselves onan open-ended task but are sufficiently cooperative to answerdirect "yes" or "no" questions honestly). Generally speaking,impression management of both kinds, whether involving guarded-ness or an effort to "look good," increases the likelihood ofseeming incongruities between Rorschach and MMPI results,whereas openness on the part of respondents is likely to promotecongruent findings. Clinicians who examine respondents moti-vated by impression management are likely to learn substantiallymore about their personality characteristics from a conjoint RIM-MMPI assessment than they would have learned from using eithertest by itself (see Ganellen, 1996b).

Even if these two patterns of RIM-MMPI results occur onlyoccasionally in clinical assessments, they send a clear message topractitioners who are trying to be helpful to their patients. There isgood reason to believe that, at least in some cases, either the RIMor the MMPI may yield apparently valid data that are neverthelessmisleading or incomplete, and there may be little way to anticipateor determine before the fact whether this will be the case. Clini-cians who ill-advisedly restrict their personality testing to eitherjust the RIM or just the MMPI will never know whether the otherinstrument would have given them complementary bits of infor-mation that could have sharpened their formulations and enhancedthe utility of their conclusions and recommendations. Such usefulcomplementarity may also involve Rorschach and MMPI findingsthat initially seem contradictory but begin to make sense after oneponders their implications in the individual case. These idiographicconsiderations in case conceptualization obtain independently of

SPECIAL SECTION: FOUNDATIONS OF THE RORSCHACH 431

whatever level of incremental validity for Rorschach variables isdemonstrated on a nomothetic basis.

These observations touch on the general topic of multimethodassessment, which has obvious implications for including the RIMin clinical test batteries but goes beyond the intended focus of thepresent commentary. Suffice it to say that, contrary to Garb's(1984) often cited indictment of test batteries for failure to dem-onstrate incremental validity for their component measures, arecent review by Meyer et al. (2001) adduced logical and empiricalconsiderations that affirm the utility of multimethod batteries as ameans of maximizing assessment validity:

We believe that there is a direct parallel between empirical researchand applied clinical practice on this issue. In research monomethodbias and nwnooperation bias are critical threats to the validity of anyinvestigation.... The optimal methodology to enhance the constructvalidity of nomothetic research consists of combining data frommultiple methods and multiple operational definitions Just aseffective nomothetic research recognizes how validity is maximizedwhen variables are measured by multiple methods, particularly whenthe methods produce meaningful discrepancies ... the quality ofidiographic assessment can be enhanced by clinicians who integratethe data from multiple methods of assessment, (p. 150)

Conclusion

Three of the five Special Series articles discussed in this com-mentary present extensive empirical evidence that the RIM hasbeen standardized, normed, made reliable, and validated in waysthat exemplify sound scientific principles for the development ofan assessment instrument. The authors of the other two articlespresent sketchy data and incomplete literature reviews on the basisof which they question the psychometric foundations of Rorschachassessment and assert that adequate validating data to demonstrateits utility have not yet been generated. Scientific debate oftenproceeds with one side arguing that something is so or somethingworks, whereas the other side argues that it is not so or does notwork. If the proponents base their arguments on accumulatingevidence of its efficacy, and the opponents base their arguments onbeing unconvinced by this evidence, there comes a time whenbeing unconvinced exceeds the boundaries of appropriate skepti-cism. Those who take issue with the abundant and compellingevidence that Rorschach assessment works very well indeed forcertain purposes for which it is intended must sooner or laterpresent equally abundant and compelling evidence that it does notwork for these purposes or else put their case to rest.

On the other hand, contemporary critics of Rorschach assess-ment are raising some appropriate concerns about the psychomet-ric status of the instrument, and those expressed in this series ofarticles include (a) the need to promote and document, if possible,accurate response coding among Rorschach practitioners as well asresearchers; (b) the importance of resolving uncertainties thatinvolve the impact of response total (R) on various summaryscores and their interpretation; and (c) the need for extendingwell-designed validity studies to many Rorschach variables thathave thus far not been adequately examined. Along with recog-nizing the legitimacy of such concerns and their implications forcontinued commitment to good Rorschach training and research,Rorschach advocates should be aware of two additional limitationsin present knowledge that also call for their attention. First, as

noted in the present article, the available normative data for Ror-schach variables may not be sufficiently representative of thecurrent United States population and does not yet fully encompasspossible cross-cultural differences. Second, very few longitudinaldata are available for either nonpatient or patient populations fromwhich life cycle developmental changes and changes in clinicalstatus over time can be adequately examined. Generalizabilityacross diverse groups of people and expected changes over timeare two areas of limited data that the RIM shares with mostpersonality assessment instruments. Closing such gaps in ourknowledge in the years ahead, aided by enhancements in dataprocessing technology and global communication, should continueto enhance the utility and luster of Rorschach assessment and otherassessment methods as well.

References

American Psychological Association (1985). Standards for educationaland psychological testing. Washington, DC: American PsychologicalAssociation.

Archer, R. P., & Gordon, R. A. (1988). MMPI and Rorschach indices ofschizophrenic and depressive diagnoses among adolescent inpatients.Journal of Personality Assessment, 52, 276—287.

Archer, R. P., & Krishnamurthy, R. (1997). MMPI-A and Rorschachindices related to depression and conduct disorder. Journal of Person-ality Assessment, 69, 517-533.

Atkinson, L. (1986). The comparative validities of the Rorschach andMMPI: A meta-analysis. Canadian Psychologist, 27, 238-247.

Bornstein, R. F. (1999). Criterion validity of objective and projectivedependency tests: A meta-analytic assessment of behavioral prediction.Psychological Assessment, II, 48-57.

Butcher, J. N., Nezami, E., & Exner, J. E., Jr. (1998). Psychologicalassessment in cross-cultural settings. In S. Kazarian & D. R. Evans(Eds.), Cultural clinical psychology (pp. 61-105). New York: Oxford.

Cohen, J. (1988). Statistical power analysis for the behavioral sciences.Hillsdale, NJ: Erlbaum.

Dawes, R. N. (1994). House of cards: Psychology and psychotherapy builton myth. New York: Free Press.

Dawes, R. N. (1999). Two methods for studying the incremental validity ofa Rorschach variable. Psychological Assessment, 11, 297-302.

Erdberg, S. P., & Shaffer, T. W. (1999, July). An international symposiumon Rorschach nonpatient data: Findings from around the world. Sym-posium conducted at the XVI International Congress of Rorschach andProjective Methods, Amsterdam.

Exner, J. E., Jr. (1991). The Rorschach: A comprehensive system. Vol. 2.Interpretation (2nd ed.). New York: Wiley.

Exner, J. E., Jr. (1993). The Rorschach: A comprehensive system. Vol. 1.Basic foundations (3rd ed.). New York: Wiley.

Exner, J. E., Jr. (1995). A Rorschach workbook for the ComprehensiveSystem (4th ed.). Asheville, NC: Rorschach Workshops.

Exner, J. E., Jr., Thomas, E. A., & Mason, B. (1985). Children's Ror-schachs: Description and prediction. Journal of Personality Assessment,49, 12-20.

Exner, J. E., Jr., & Weiner, I. B. (1995). The Rorschach: A comprehensivesystem. Vol. 3. Assessment of children and adolescents (2nd ed.). NewYork: Wiley.

Finn, S. E. (1996). Assessment feedback integrating MMPI-2 and Ror-schach findings. Journal of Personality Assessment, 67, 543-557.

Gacono, C. R., & Meloy, J. R. (1994). The Rorschach assessment ofaggressive and psychopathic personalities. Hillsdale, NJ: Erlbaum.

Ganellen, R. J. (1996a). Exploring MMPI—Rorschach relationships. Jour-nal of Personality Assessment, 67, 529-542.

Ganellen, R. J. (1996b). Integrating the Rorschach and the MMPI inpersonality assessment. Mahwah, NJ: Erlbaum.

432 WEINER

Garb, H. N. (1984). The incremental validity of information used inpersonality assessment. Clinical Psychology Review, 4, 641-655.

Garb, H. N. (1998). Studying the clinician: Judgment research and psy-chological assessment. Washington, DC: American Psychological As-sociation.

Garb, H. N., Florio, C. M., & Grove, W. M. (1998). The validity of theRorschach and the Minnesota Multiphasic Personality Inventory: Re-sults from meta-analyses. Psychological Science, 9, 402-404.

Garb, H. N., Florio, C. M., & Grove, W. M. (1999). The Rorschachcontroversy: Reply to Parker, Hunsley, and Hanson. Psychological Sci-ence, 10, 293-294.

Hiller, J. B., Rosenthal, R., Bernstein, R. F., Berry, D. T. R., & Brunell-Neuleib, S. (1999). A comparative meta-analysis of Rorschach andMMPI validity. Psychological Assessment, 11, 278-296.

Hilsenroth, M. J. (2000). Rorschach test. In A. E. Kazdin (Ed.), Encyclo-pedia of psychology (pp. 117-119). Washington, DC: American Psy-chological Association.

Hilsenroth, M. J., Fowler, J. C., Padawer, J. R., & Handler, L. (1997).Narcissism in the Rorschach revisited: Some reflections on empiricaldata. Psychological Assessment, 9, 113-121.

Hunsley, J., & Bailey, J. M. (1999). The clinical utility of the Rorschach:Unfulfilled promises and an uncertain future. Psychological Assessment,11, 266-277.

Masling, J. M. (1997). On the nature and utility of projective tests andobjective tests. Journal of Personality Assessment, 69, 257-270.

McCann, J. T. (1998). Defending the Rorschach in court: An analysis ofadmissibility using legal and professional standards. Journal of Person-ality Assessment, 70, 135-144.

McClelland, D. C., Koestner, R., & Weinberger, J. (1989). How doself-attributed and implicit motives differ? Psychological Review, 96,690-702.

McDowell, C., & Acklin, M. W. (1996). Standardizing procedures forcalculating Rorschach interrater reliability. Journal of Personality As-sessment, 66, 308-320.

Meloy, J. R., Acklin, M. W., Gacono, C. B., Murray, J. F., & Peterson,C. A. (Eds.). (1997). Contemporary Rorschach interpretation. Mahwah,NJ: Erlbaum.

Meyer, G. J. (1997a) Assessing reliability: Critical corrections for a criticalexamination of the Rorschach Comprehensive System. PsychologicalAssessment, 9, 480-489.

Meyer, G. J. (1997b). On the integration of personality assessment meth-ods: The Rorschach and MMPI. Journal of Personality Assessment, 68,297-330.

Meyer, G. J. (1997c). Thinking clearly about reliability: More criticalcorrections regarding the Rorschach Comprehensive System. Psycho-logical Assessment, 9, 495-498.

Meyer, G. J. (1998). Error in research and assessment data with an erratumfor Meyer (1993). Journal of Personality Assessment, 71, 195-211.

Meyer, G. J. (1999a). Convergent validity of the MMPI and Rorschachscales: An extension using profile scores to define response and char-acter styles on both methods and a reexamination of simple Rorschachresponse frequency. Journal of Personality Assessment, 72, 1-35.

Meyer, G. J. (1999b). Introduction to the special series on the utility of theRorschach for clinical assessment. Psychological Assessment, 11, 235-239.

Meyer, G. J., Finn, S. E., Eyde, L. D., Kay, G. G., Moreland, K. L., Dies,

R. R., Eisman, E. J., & Kubiszyn, T. W. (2001). Psychological testingand psychological assessment: A review of evidence and issues. Amer-ican Psychologist, 56, 128-165.

Meyer, G. J., & Handler, L. (1997). The ability of the Rorschach to predictsubsequent outcome: Meta-analysis of the Rorschach Prognostic RatingScale. Journal of Personality Assessment, 69, 1-38.

Parker, K. C. H., Hanson, R., & Hunsley, J. (1988). MMPI, Rorschach, andWAIS: A meta-analytic comparison of reliability, stability, and validity.Psychological Bulletin, 103, 367-373.

Parker, K. C. H., Hunsley, J., & Hanson, R. K. (1999). Old wine from oldskins sometimes tastes like vinegar A response to Garb, Florio, andGrove. Psychological Science, 10, 291-292.

Perry, W., Geyer, M. A., & Braff, D. L. (1999). Sensorimotor gating andthought disturbance measured in close temporal proximity in schizo-phrenic patients. Archives of General Psychiatry, 56, 277-281.

Perry, W., Viglione, D. J., & Braff, D. (1992). The Ego Impairment Indexand schizophrenia: A validation study. Journal of Personality Assess-ment. 59, 165-175.

Rorschach, H. (1942). Psychodiagnostics: A diagnostic test based onperception. Bern, Switzerland: Hans Huber. (Original work published in1921)

Smith, B. L. (1994). Object relations theory and the integration of empir-ical and psychoanalytic approaches to Rorschach interpretation. Ror-schachiana, 19, 61—77.

Strieker, G., & Gold, J. R. (1999). The Rorschach: Towards a nomotheti-cally based, idiographically applicable, configurational model. Psycho-logical Assessment, 11, 240-250.

Viglione, D. J. (1999) A review of recent research addressing the utility ofthe Rorschach. Psychological Assessment, 11, 251-265.

Weiner, I. B. (1993). Clinical considerations in the conjoint use of theRorschach and the MMPI. Journal of Personality Assessment, 60, 142-148.

Weiner, I. B. (1995). Methodological considerations in Rorschach re-search. Psychological Assessment, 7, 330-337.

Weiner, I. B. (1996). Some observations on the validity of the RorschachInkblot Method. Psychological Assessment, 8, 206-213.

Weiner, I. B. (1998a). Principles of Rorschach interpretation. Mahwah,NJ: Erlbaum.

Weiner, I. B. (1998b). Teaching the Rorschach Comprehensive System. InL. Handler & M. J. Hilsenroth (Eds.), Teaching and learning personalityassessment (pp. 215-233). Mahwah, NJ: Erlbaum.

Weiner, I. B. (1999). Contemporary perspectives on Rorschach assessment.European Journal of Psychological Assessment, 15, 78 -96.

Wenar, C., & Curtis, K. M. (1991). The validity of the Rorschach forassessing cognitive and affective changes. Journal of Personality As-sessment, 57, 291-308.

Wood, J. M., Nezworski, M. T., & Stejskal, W. J. (1996). The Compre-hensive System for the Rorschach: A critical examination. PsychologicalScience, 7, 3-10.

Wood, J. M., Nezworski, M. T., & Stejskal, W. J. (1997). The reliability ofthe Comprehensive System: A comment on Meyer (1997). Psychologi-cal Assessment, 9, 490-494.

Received May 24, 1999Revision received October 11, 1999

Accepted May 10, 2001 •