A MODERATOR VARIABLE APPROACH TO &IMP1 VALIDITY DAVID OSBORNE*

hf ayo Clinic and hfayo Foundation, Rochester, Minnesota

PROBLEM The suggestion made by Rosanoff (’), that sources of error within a test be

measured and corrected, led to the development of validity scales for personality inventories. The “No Count” of Humm and Wadsworth(6) and the MMPI validity scales L, F, and K ( * ) are examples of the type of internal validity indicator sug- gested by Rosanoff.

Saunders@) has pointed to a type of “moderator’) variable that is related to individual errors of measurement. Scores on a moderator variable allow the pre- diction of a person’s error of measurement and the tailoring of prediction strategy accordingly. Ghiselli (a ) has called this phenomenon “prediction of predictability”. A moderator variable indicates how well a given test is predicting for a given individual. Demonstrations of moderating effects have been provided by Abelson(l), Prederickson and Melville ( I ) , and Steinemann ( l o ). Smith and Lanyon (9) attempted to develop a moderator scale from AIMPI items for use in predicting juvenile probation violations from a base expectancy table but were not able to develop a scale that held up on cross validation. In spite of their failure to develop a useful scale they commented, “. . . in view of the success of the moderator-variable ap- proach in personnel and educational prediction, research should be continued in efforts to apply this approach to clinical situations.”

This study examined sources of prediction error in the MMPI using the mod- erator variable approach. An attempt was made to develop a validity scale that would divide MMPI profiles into two groups: A group of profiles that yield accurate predictions about the subjects and a group that yield inaccurate predictions.

METHOD Subjects. The Ss were 56 male and 90 female psychiatric inpatients at a

university medical center. The average age was 36.5 years, range 15 to 71 years. In order for a patient to be included in this study, his admission MMPI had to meet the following criteria: (1) P raw score less than or equal to 10; (2) L raw score less than or equal to 10 (T score less than or equal to 70) ; (3) F raw score less than or equal to 21 (T score less than or equal to 90) ; (4) K raw score less than or equal to 23 (T score less than or equal to 70) ; and (5) F minus K index less than or equal to 11. Profiles with scores above these cutoff points were eliminated so that any prediction error in the investigation would be an error that the validity scales failed to identify.

M M P I Administration. The card form of the MMPI was administered individually or in small groups, an average of two days after the patient was admit- ted to the hospital, with a maximal time span of 21 days. Ninety-five percent of the Ss took the MMPI within one week of admission.

Judging Procedure. A list of 31 symptoms was compiled from the admission notes of the patients’ hospital charts. These admission notes had been made before the results of the MMPI were available to the admitting physician. Any symptom that appeared in the admission notes of five or more Ss was included on the list. Two experienced clinical psychologists were asked to examine the MMPI profile and to check the symptoms that they would expect the patient to have on admission. They were to check no more than five symptoms; however, they could check fewer than five. Each profile contained scores on scales L, F, K, Hs, D , Hy, P d , M f ,

*J. N. Butcher gave valuable advice and sup ort i i i this study. John P. Braiitiier, Alan H. Roberts, and Lloyd K. Sines acted as clinical judges for tRe iiivestigatioii.

,% AIODERATOR VARIABLE APPROACH TO JIMPI VALIDITT 187

TABLE 1. MMPI ITEMS THAT IDENTIFY TEST “MIMES”

Scored Item no. Item response

)

!)

43 74

76 79 89

101 107 I38 1 4% 138 1 .iY 160 213 319 236 242 274 301 303 335 36 1 :{7 1

:179 :I84 407 114 442

.iK2

.-BOG

.i26

. i 3 3

3 6 .-A0 - - - . ).I:>

.i.i8

.560

I am easily awakened by noise. I am about as able to work as I ever was. My sleep is fit.ful and disturbed. I have often wished I were a girl. (Or if you are a girl) I have never beeii

sorry that I am a girl. Most of the time I feel blue. My feelings are not easily hurt. It takes ti lot of argument to convince inost people of the truth. I helieve women ought t,o have as much sexual freedom as meii. I am happy most of t.he time. Criticism or scolding hurts me terribly. I certainly feel useless a t tirnw. I cry easily. I cannot understand what I read as well as I used to. I have never felt, better in my life than I do now. I hnve used alcohol excessively. I think I would like the work of a building contractor. I brood n great deal. I helieve I am n o more nervous than most others. My eyesight is :w good as it has beeii for years. Life is n strain for me much of the time. I am so toiichy o n some subjects that I can’t talk about them. I cannot keep my mind on one thiiig. I am inrlined t.o take things hard. I am not usually self-conscious. I very seldom have spells of the blues. I am unable to tell anyone all about myself. I nni u3ually calm and not. easily upset. I am apt to take disappointments so keenly that I can’t put them out of my m I have had periods in which I lost sleep over worry. I like to let. people know where I stand on things. I am R high-strung person. The future w m s hopeless to me. My mouth feels dry almost all of the time. It. makes me angry to have people hurry me. My face ha5 never been paralyzed. I somet.imes feel that, I am about to go to pieces. A large number of people are guilty of bad sexual conduct. I am greatly bothered by forgettirlg where I put things.

iind .

F ’L‘ F

F F r r

‘I‘ ‘I‘ T F 1; F F T T T F T T F F F F T T F T F F T F F F F T F T F

48s DAVID OSBOHNE

Mean

Development Cross-validation Sample Sample

13.58 25.36 13.98 20.33

48s DAVID OSBOHNE

Pa, Pt, Sc, Ma, Si, and E s , as well as the age and sex of the patient. The profiles were then divided into a group that yielded relatively accurate predictions (hits) and a group that yielded relatively inaccurate predictions (misses), by comparing the MMPI interpretation to the admission note.

A random sample of 30 profiles was given to both judges for the purpose of assessing the reliability of the judging procedure. The remaining 116 profiles were divided between the judges. A product moment correlation coefficient was cal- culated from the judges’ scores on the 30 reliability profiles to measure agreement between observers.

After “hit” and “miss” groups were formed, each was randomly divided into a development sample and a cross-validation sample to produce four groups: (1) hits-development sample ( N = 40); (2) misses-development sample (N = 33) ; (3) hits-cross-validation sample ( N = 40) ; and (4) misses-cross-validation sample (N = 33).

Item Analysis. An item analysis of the XIRIPI was undertaken to select items that were endorsed in significantly different proportions by the hits and the misses of the development sample. The 2 statistic(a) was used to select items that discriminated between the two groups a t the 5% level.

The significant items were assembled into a scale. Each S in the cross-validation sample was scored on the scale in which one point was scored each time the S answered a significant item in the same direction as the “miss” group in the development sample. A high score on the scale meant that the S endorsed items in a manner similar to that of Ss whose MMPIs yielded relatively inaccurate predictions. As a first and necessary test of the scale, mean scores were calculated for the “hit” and “miss” groups of the cross-validation sample, and the 2 test(11) of significance was applied.

RESULTS The product moment correlation coefficient, calculated from the number of

symptoms correctly predicted for each profile, as a measure of agreement between observers was .72 (p < 0.01), thus indicating good agreement between the judges in the accuracy of their MMPI interpretations.

The 38 MMPI items listed in Table 1 discriminated between the hits arid the misses of the development sample a t the 5% level of significance. The indicated scoring direction is used to identify profiles that are more likely to yield inaccurate predictions. The scale was named the moderator scale and was given the desig- nation Mo. Twenty-five of the items were scored on other IllA4PI clinical and validity scales, most of them in the opposite direction. The only item that was scored on a clinical scale in the same direction as on the moderator scale was “I have used alcohol excessively” (true).

Cross Validation.

TABLE 2. MEAN 1 t . i ~ SCORES ON Scam U o FOR DEVELOPMENT AND CROSS-VALIDATION SAMPUS

The MRlPIs of the Ss in the cross-validation sample were scored on scale Mo. Scores for the “hit” and “miss” groups in the development and the cross-validation samples are found in Table 2. A raw score cutoff point of 21 was found to be most

A MODERATOR VARIABLE APPROACH TO YMPI VALIDITY 489

Scale

T A U = 3. CLASSIFICATION OF ‘‘HITS’’ AND “MIBBEB” BY MODERATOR CUTOFF &orup OF 21

Moderator raw score

21 O G b r LeM th.n 21

Criterion Category (Hit)

_ _ _ ~ .

Hit (N = 80) M k ( N = 86)

Mean SD

Mies 18 15 3 Hit 37

Phi coefficient = .44 ( p < .001). __ _-

efficient for identifying a large number of miseeel while misclassifying the smallest number of hits (Table 3). The comlations between the moderator scale and other MMPI scales are presented in Table 4.

T.4~m 4. PRODIJC~-MOUONT ( ~ W B L A T I O N S OF MODBEATOR SCALE W m NON-K

COIUWXED MMPI &A- (N - 146)

Scale r

L .a* F - .14 K .21- H.3 - .51* D - .79*

- ,57* - .34* - .51* Pa

Pt - .78* sc - .w M a .08 Si -.a*

%!

-________-_---- ‘ P < .Ol;**P < .05.

490 DAVID OSBORNE

DISCUSSIOX The low magnitude of the correlation between the moderator scale and the

validity scales and the absence of significant differences between the groups on the validity scales suggest that the moderator scale identifies a source of error in the MMPI not identified by scales L, F , and K .

Of the 38 items discriminating between the hits and the misses in the item analysis, 33 can be construed as denials of problems or pathology when scored to identify misses. Only 1 of the 38 items was scored in the same direction on both a clinical scale and the moderator scale. The results of the item analysis suggest that the patients who are test misses attempted to deny pathology in their responses. To approach the results from another direction, the patients who are test hits endorsed items indicative of pathology and personal problems.

The question of the source of the difference in profile elevation has not been treated in this investigation. An actual difference in degree of pathology may exist between the hits and the misses, or perhaps the difference could be accounted for by the operation of a response set. The misses, for example, may be answering items in a socially desirable direction rather than revealing their symptoms.

CONCLUSIONS In an inpatient psychiatric setting, elevated hIMPI profiles are more likely

to yield accurate predictions about the patient than are low profiles. Scale M o is proposed as a new validity scale for the MMPI to be used in identifying the new source of prediction error. The raw score cutoff point of 21, which was used in this investigation, may not be optimal in other settings; therefore, i t is suggested that anyone using scale M o develop a cutoff point appropriate to his own population.

REFERENCES 1 . ABELSON, It. P. Sex differelices in predictability of college grades. h’du .. l’sychol. Meas., 1932,

2. DAHLSTROM, W. G. and WELSH, G . S . (Eds.) AIMPZ Handbook: A Guide to Use in Clinical

3. EDWARDS, A. L. Ezperimenlal Design in Psychological Research. New York: Holt, Kinehart and

4. FREDRICKSON, N. and MELVILLE, S. L). Differential predictability in the use of test scores.

5. GHISELLI, E. E. The prediction of predictability. Educ. Psychol. Aleas . , 1960, 20, 3-8. 6. HUMM, D. G. and WADSWORTH, G. W., JR. The Humm-Wadsworth temperament scale. .3mer.

7. ROSANOFF, A. J. Cited by Humm, L). G. and Wadsworth, G. W., Jr. ( 6 )

8. SAUNDERS, D. R. Moderator variables in prediction. Educ. Psychol., 19.i6, 16, 209-222. 9. SMITH, J. and LANYON, R. I. Prediction of juvenile probation violators. J . consult. clin. Psychol.,

1.9, 638-644.

P r d i c e and Research. Minneapolis: University of Minnesota Press, 1960.

Winston, 1963.

Educ. Psychol. Meas., 1954,14, 647-656.

J . Psydricrt., 1935, 9.9, 163-200.

1968. 3.9. 54-58. ~ - , -~ -~ 10,- &WINEMANN, J. H. Use of n logicnlly related predictor in determining intragroup differential

11. WINER, $: J. (Ed.) Statistical Princzples in Ezperimdal Design. New York: McGraw-Hill, predictabilit . J . appl. Psychol., 1964,48, 336-338.

1962.