77-1D- Ai3 889 AN EVALUATION OF ONE- AND THREE-PRAETER LOGISTIC i/l

TAILORED TESING FROCE (U) AMERICAN COLL TESTING

PROGRAM IOWlA CITY IA R.ENTE PROGRAMS D.

UNCLASSIFIED R L IICKINLEY ET AL.AG R- 8ib-OR F/ 12/ N

%"it' i

11111 - ..-.2..-.

1111Eu _ L 2.5 7

-....

125 ___ 116

MICROCOPY RESOLUTION TEST CHARTNATBONAL BUREAU OF STANDARDS.663-A(

... '...'-J-,~o"

%,,- %,' %7. -' ',,, '% ' ., . - -' -' -, . .. : . : ; -' .' .'i . , . . : .. -, -, ,::, -, -. .- . -; - . -.- - --. - - - . . . . . . .-p,,.,,,,;,.. .,,,,,,-", ,,, . ... ,.,. ,. , .. . ..... .. ,. , ., ..... .•.. . . .. .-... . .. ..

0o An Evaluation of One- and Three-Parametert . Logistic Tailored Testing Procedures

for Use with Small Item Pools

, Robert L McKinleyand

car. Mark D. Reckase

Research Report ONR83-1

August 1983

.4

The American College Testing Program Q T'l -Resident Programs Department F E EG'

Iowa City, Iowa 52243 B 1 314

a3-

L.J Prepared under Contract No. N00014-82-K07f6 ""4 with the Personnel and Training Research Programs

a.. Psychological Sziences DivisionOffice of Naval Research

5Approved for public release, distribution unlimited.Reproduction in whole or in part is permitted forany purpose of the United States Government.

84 02 13 017.

I SECURITY CLASSIFICATION OF THIS PAGE flior Does Entered)

REPORT DOCUMENTATION PAGE READ INSTRUCTIONSREPORT__ DOCUMENTATIONPAGE_ BEFORE COMPLETING FORMI 8REPORT NUMBER GOVT ACCESSION NO. 3. RECIPIENT'S CATALOG NUMBER

4 TITLE (ad Subtitle) S. TYPE OF REPORT & PERIOD COVERED

An Evaluation of One- and Three-ParameterLogistic Tailored Testing Procedures Technical Reportfor use with Small Item Pools 6 PERFORMING ORG. REPORT NUMBER

7. AuTHOR(e) S. CONTRACT OR GRANT NUMBER(*)

Robert L. McKinley and N00014-82-K0716

Mark D. Reckase

9 PERFORMING ORGANIZATION NAME AND ADDRESS 10. PROGRAM ELEMENT. PROJECT. TASKAREA & WORK UNIT NUMBERS

The American College Testing Program P.E.: 61153N Proj.: RR04,-04P.O. Box 168 r.A.: 042-04-01Iowa City. IA 52243 WV_. NR]Sn-499,, CONTROLLING OFFICE NAME AND ADDRESS ,2 REPORT DATE

Personnel and Training Research Programs August 1983Office of Naval Research 13 N UME O "0 F PA GESAr" ino i--ii-in V I ",oi ' 99917 24__ _ __ _ _ _ __ _ _ _ _

4 MONIr RING'AGNC NAME a ADDRESS(It differenl from Controllind Office) IS. SECURITY CLASS. (of tl, report)

unclassified1 l~ I. DECLASSIFIrCATION/DOWNGRAOING

SCHEDULE

IS. DISTRIBUTION STATEMENT (of tile Report)

Approved for public release, distribution unlimited. Reproduction inwhole or part is permitted for any purpose of the United States government.

17. DISTRIBUTION STATEMENT (of the abstract eanered In Block 20, It lfferent from Report)

19. SUPPLEMENTARY NOTES

It. KEY WORDS (Continue a Irvere i e If necessmy and identify by block number)

Latent trait models

Tailored TestingComputerized Testing

' .- 20. ABSTRACT (Continue on revre slide It necessary and Identify by block Mibor)

An Evaluation of One- and Three-Parameter

Logistic Tailored Testing Proceduresfor use with Small Item Pools

ABSTRACT

DD ,A01r 1473 EDITION O,' NOV, S OBSOLETE~~~~~~~~~~~~~~S IN 0 102- LF-014- 6601 r-, ,,,. ,- e, , ,, , ,, r,, .. .. =, , .,

P31 1W ITTIEd.-N w6-V .1- .97

SECURITY CLASSIFICATION OF THIS PAGE (Wha Dae Entered

, A two-stage study was conducted to compare the ability estimatesyielded by tailored testing procedures based on the one-parameterlogistic (IPL) and three-parameter logistic (3PL) models. The firststage of the study employed real data, while the second stage employedsimulated data. In the first stage, response data for 3000 examineeswere obtained for the 40 item ACT Assessment Mathematics Usage subtest.The first 2000 cases were used to obtain item parameter estimates forboth models. Using these estimates, 1PL and 3PL tailored tests weresimulated using the response data for the remaining 1000 cases. Bothtailored testing procedures employed maximum likelihood ability estimationand maximum information item selection procedures. The two sets of abilityestimates were then compared. In the second stage, response data for3000 cases were simulated using the 3PL item parameter estimates fromthe first stage as true parameters. True abilities were selected fromthe standard normal distribution. The first 2000 cases were used for1PL and 3PL calibration of the items, and the remaining 1000 caseswere used to simulate 1PL and 3PL tailored tests. The two sets ofability estimates were compared to each other and to the true abilityparameters. Results of both stages of the study indicated that the1PL and 3PL tailored tests yielded highly correlated ability estimates,and there was no apparent advantage in terms of ability estimation tousing one of the models over the other. Because the 1PL procedure wasless expensive to use, it was the recommended model for this application.

Avoesson ForNTIS GRA&IDTIC TAB

-UnaouneeA"i J-usttif icat!,-. ..

Distribution/

sAvailability CodesAvail and/or

D 'iS t Special

.4

.4

S/N 0102"- L 0 14-6601

SECU ITY CLASSiFICATION OF ThIS PAGIErWtmn Vet Entered)

CONTENTS

PageIntroduction..........................

Comparison of lPL and 3PL Tailored Testing Procedures . 1

Method ........................... 3Models.........................3Estimation Programs...................3Tailored Testing Procedures ............... 3Design.........................5Data..........................5Analyses........................5

Results..........................6Real Data Analyses...................6

Item Pool Calibration...............6Ability Estimates................8Average Test Length................11Nonconvergence .................. 11

Simulation Data Analyses................12Item Pool Calibration .............. 12Ability Estimates.................16Average Test Length................18Nonconvergence .................. 18

Discussion.........................18The Application ..................... 1Real Data Analyses..................20

Item Pool Calibration .............. 20Ability estimates.................20Average Test Length ............... 20Nonconvergence..................20

Simulation Data Analyses................21Item Pool Calibration .............. 21Ability Estimates.................21Average Test Length ............... 21Nonconvergence..................21

Summary and Conclusions..................21

References..........................23

An Evaluation of One- and Three-ParameterLogistic Tailored Testing Procedures

for use with Small Item Pools

Tailored testing has shown considerable promise as analternative to conventional paper-and-pencil testing, butbefore it can be implemented on a widescale basis, a numberof issues must be addressed. Tailored testing proceduresinvolve a number of complex components, and there are oftena number of alternatives which may be chosen for each.Although there has been considerable research conducted inthis area, it is still unclear which of the many alternativecomponents should be used in any particular application.For instance, one important component of tailored testing isthe item response theory(IRT) model upon which the procedureis to be based. There are numerous IRT models, several ofwhich have been.proposed for use in tailored testing. Thepurpose of this study was to compare tailored testingprocedures based on two of the most popular IRT models, theone-parameter lo!tistic (1PL) and three-parameter logistic(3PL) models, to determine whether one of the two models ispreferable to the other in a tailored achievement testingsetting. The tailored testing procedures based on the lPLand 3PL models were compared on the basis of the abilityestimates which were yielded by the procedures. Beforereporting the results of the study, it may be helpful toreview previous research comparing tailored testingprocedures based on these two models.

Comparisons of IPL and 3PL Tailored Testing Procedures

Several studies have been conducted to compare the use ofthe 1PL and 3PL models for tailored testing. One suchstudy, reported by Koch and Reckase (1978), was a directcomparison of 1PL and 3PL tailored testing procedures in anapplication to vocabulary measurement. Both proceduresemployed maximum likelihood ability estimation techniques,and in both procedures items were selected to maximize theinformation function at the current ability estimate. Theresults of this study indicated that both models could besuccessfully applied to vocabulary ability measurement. The3PL procedure had a slightly higher reliability (a crossbetween test-retest and equivalent forms reliabilities) thanthe IPL procedure (r = .77 for the 3PL procedure, r = .61for the 1PL procedure). However, the 3PL procedure failedto converge to ability estimates in nearly one third of thecases, while nonconvergence was not a serious problem withthe 1PL procedure.

W~~ 7~~A ~ 7 ~Y 7 r '. -- . .-.--

-2-

In a second study, reported by Koch and Reckase (1979),1PL and 3PL tailored testing procedures were applied to amultidimensional achievement test. Results of this studyindicated very poor performance for both procedures,primarily due to small sample sizes, poor linkingprocedures, and poor selection of the stepsize and initialability estimates for the maximum likelihood estimationprocedure.

A study reported by McKinley and Reckase (1980) attemptedto correct the problems encountered in the Koch and Reckasestudies. Close attention was paid to appropriate itemparameter linking and selection of the operatingcharacteristics of the procedures. The results of thisstudy indicated that both models could be quite successfullyapplied to tailored achievement testing if correctly

4implemented. Both IPL and 3PL reliabilities were higherthan the reliability of a classroom test over the samematerial. The 3PL procedure yielded better fit to the datathan the 1PL procedure, and it also yielded higher test

*information than the IPL procedure. This study concludedthat for tailored achievement testing the 3PL model was themodel of choice. However, the test used in this study was

* highly multidimensional. It is unclear how generalizablethe results are to less multidimensional achievement test.

Urry (1970, 1977) also concluded that the 3PL model wasthe model of choice. Through a series of simulation studiesUrry found that tailored testing becomes less effective whena model with an insufficient number of parameters is used.He concluded that construct valdity decreases as a functionof the degree of degeneracy of the model, and the IPL modelwas particularly inappropriate for use with multiple-choiceitems because it did not portray multiple-choice responsedata with fidelity (Urry, 1977).

This review of previous research indicates that ifcareful attention is paid to all components of the tailored

-testing procedure, both 1PL and 3PL tailored testing can besuccessful. The 3PL model tends to yield higherreliabilities and test information than the lPL procedure,but is more prone to complications such as nonconvergence.It is also indicated that the 3PL model yields better fit tomultidimensional data. Thus, the results of these studiestend to favor the 3PL model. Of course, these results wereobtained using relatively large item pools. It is unclearfrom these studies what results would be obtained usingsmaller item pools. The purpose of this study was tocompare the 1PL and 3PL models in a tailored achievementtesting application for which a relatively small item pool

-3-

is available.

Method

Models

The two models selected for this study were the one-parameter logistic (IPL) and the three-parameter logistic(3PL) models. The IPL model is given by

exp( (8.-b.)x. )

P(x ij) = I

1 1+exp(8j-bi)

where 6. is the ability parameter for examinee j, bi is the

difficulty parameter for item i, xii is the observed score

(0 or 1) on item i for examinee j, and P(x ij) is the

probability of response x.. to item i by examinee j. The

3PL model is given by

~exp(Da.(8.-b.))

P(X. =1) = c.+(l-c.) J

1+exp(Dai(8.-b.))

where ci is the pseudo-guessing parameter for item i, ai is

the discrimination parameter for item i, where Pi(Bj) is the

probability of a correct response to item i by examinee j,and the remaining terms are as previously defined.

Estimation Programs

For both the lPL and the 3PL models parameters wereestimated using the LOGIST program (Wingersky, Barton, andLord, 1982). For the lPL model the pseudo-guessingparameter was held fixed at 0.0. The discriminationparameter was held fixed at a value computed by the LOGISTprogram. To check the IPL estimates obtained from LOGIST,they were compared to parameter estimates obtained for thesame data using the MAX program (Wright and Panchapakesan,1969), which was designed for use with the IPL model. Sincethe results obtained from the two programs were almostidentical, LOGIST was used throughout the study. The LOGISTprogram was used for both models in order to avoid problemsdue to different parameter estimate scales. For both modelsthe scales were based on the ability estimate distributions.

Tailored Testing Procedures

Tailored testing procedures have three main components:an item selection routine, an ability estimation technique,

-4-

and a stopning rule. In this study both the 1PL and 3PLprocedures selected items to maximize the value of theinformation function (Birnbaum, 1968) at the most recentability estimate. The information for each item at theexaminee's current ability estimate was computed, and theitem with the greatest information at that ability estimatewas administered, with the provision that the informationhad to be greater than 0.226 for the 1PL procedure and 0.450for the 3PL procedure. These values were selected on thebasis of several trial runs. They were selected so as toyield approximately equal average test lengths for the twomodels. For both procedures 20 items was the maximum testlength allowed.

Prior to testing initial estimates of ability wereassigned to set the starting points in the item pool. Theinitial ability estimates for this.study were set to be0.221 for the 1PL procedure and 0.420 for the 3PL procedure.These values represent difficulty values near the medians ofthe item pool difficulty parameter distributions. The firstitem was then selected to maximize information at theinitial ability estimate. The response of the examinee tothat item was then simulated in the following manner. Forthe first part of the study, response data came from a fixedlength, non-tailored test comprised of all the items in thepool. These items had been administered in paper and pencilform to all of the examinees used in this study. Anexaminee's response to an item in the tailored tests was theactual response of the examinee to the item on the paper andpencil test. For the second part of the study, simulatedresponse data were generated for each examinee for each itemin the pool. These data were generated according to the 3PLmodel using the 3PL item parameter estimates obtained forthe real response data and examinee abilities selected atrandom from a standard normal distribution. These responseswere used regardless of whether a 1PL or 3PL based tailoredtest were used.

Once the response by an examinee to an item had beenobtained, a new estimate of ability was computed by adding afixed stepsize to the old ability estimate if the responsewere correct, and by subtracting a fixed stepsize if theresponse were incorrect. This fixed stepsize procedure wasused until a maximum likelihood ability estimate could beobtained (i.e., when both correct and incorrect responseswere obtained). The stepsize used was 0.300 for both

procedures. Each new item was selected to maxim-ize theinformation at the new ability estimate, with therestriction that no item could be used more than once.

!"

.J * .. -- . - .-. . ,. . - - . - .I.I

-5-

Two stopping rules were used for the tailored testingprocedures. The tests were terminated when there were noitems left in the item pool with information at the currentability estimate greater than the minimum specified above,or when 20 items had been administered.

Design

This study employed a two-stage design--one involving theuse of real data, and one involving simulated data. In thefirst stage of the study, response data were obtained for alarge sample on a relatively short paper and pencil test.Part of the large sample was then used to calibrate theitems on the test using both the IPL and 3PL models. Usingthe resulting item parameter estimates, lPL and 3PL tailoredtests were simulated for the examinees not included in thecalibration sample. The responses by the examinees to theitems in the tailored tests were the same responses theymade to the items when taking the paper and pencil test.

-' In the second stage of the study, the item parameterestimates obtained from the 3PL calibration of the paper andpencil test were used as true parameters, along with thetrue abilities selected at random from the standard normaldistribution, to generate simulated response data to fit the3PL model. Data were generated for a large sample for allthe items from the paper and pencil test. The procedure

used for the real data part of the study was then repeatedusing these simulated data.

Data

For the real data part of the study, response data forthe 40 item Mathematics Usage subtest of the ACT Assessment

(The American College Testing Program, 1982) were obtainedfor 3000 cases from the October, 1982 administration of theACT Assessment (Form 23B). For the second stage of thestudy, data were simulated for 40 items and 3000 cases. For

C. both stages, then, rather small item pools were used.

Analyses

The analyses performed in this study consisted primarilyof computing and comparing correlations. For both the realand the simulation data, the 1PL and 3PL tailored testability estimates were compared by computing the correlation

- between them. For the simulation data the two sets ofability estimates obtained from the tailored tests were alsocompared to the true abilities used to generate the data.Again, the comparisons were performed using correlations.

.44 - • • o-.-'.- '. .- -... ...

-6- .1

Results4.

Real Data Analyses

Item Pool Calibration The first analysis performed on thereal data was the calibration of the items for use as atailored testing item pool. The calibration of the items,which was based on response data for the first 2000examinees, was performed three different ways. The firsttwo calibrations were performed for the IPL model using theLOGIST and MAX programs while the third was performed forthe 3PL model using LOGIST. The MAX and LOGIST IPL itemdifficulty parameter estimates had a correlation of 0.999,as did the ability estimates obtained from the two programs.This comparison was performed in order to determine whetherthe LOGIST program could be used for both models throughout

*the study. These findings indicated that it could, thussimplifying the problem of placing the estimates from thetwo models on the same scale.

The item parameter estimate distributions obtained forthe two models using LOGIST are shown in Figure 1. Thesedistributions are summarized by the statistics shown inTable 1. As can be seen, most of the 3PL discriminationparameter estimates were .60 or higher, so most of the itemswere of fairly high quality. From the 3PL difficultyparameter estimate distribution, however, it can be seenthat the items are appropriate only for a limited range ofability, since most of the item difficulty estimates fall inthe range from -1.0 to 1.75. Most of guessing parameterestimates are .3 or less, with only two items havingguessing parameter estimates greater than .3. From thesedata it would appear that these items actually form a fairlyhigh quality item pool for tailored testing, except for thelimitation on the range of difficulty.

For the lPL model, the LOGIST program assigned to allitems a discrimination value of 0.561. The pseudo-guessing

* parameter was, of course, 0.0. The IPL difficulty parameter*. estimate distribution is somewhat different from the 3PL

difficulty distribution although the two sets of estimateshad a correlation of .88, with the biggest difference beinga shift downward of the bulk of the estimates for the 1PLmodel. Most of the difficulty parameter estimates fallwithin the same range as for the 3PL model, but thereappears to be a shift toward the negative end of that range.Still, for that range the items form an item pool of fairlyhigh quality.

%F

J.1

J-

-7S 07 -7 %1 1.- -7 07 W.- J. T. 12. -7 17a ',. %. %- -X .

Figure 1

The 1PL and 3PL Item Parameter EstimateFrequency Distributions for the Real Data

FL A

0

0.0 i5 1.0 1.5 2.0 2.5 3.0

3PL B

09

-4.S5 30 - . "S30'.

~ - *'I. ~ ~ . * - a . .

-8-

Table 1Descriptive Statistics of Item Parameter

Estimates for the Real Data

IPL 3PL

Statisticb a b c

Mean 0.03 0.98 0.46 0.17Median 0.22 0.90 0.41 0.16S.D. 0.91 0.34 1.10 0.08Skewness -0.24 0.40 -0.20 1.14Kurtosis 0.19 -0.04 0.99 1.19Low Value -2.07 0.31 -2.12 0.08High Value 2.04 1.81 3.15 0.41

Figure 2 shows the test information function for the itempool based on the 1PL item parameter estimates, while Figure3 shows the test information function based )n the 3PLestimates. As can be seen from Figure 3, the 3PL curve isnegatively skewed, and is centered around 1.0, thus yieldingmore information for the positive end of the ability scale.The 1PL curve, on the other hand, is not skewed, and iscentered around 0.2. It would appear from this, then, thatthe 1PL item parameter estimates are appropriate for a widerrange for ability than the 3PL estimates are. Of course,the ability scales are not exactly comparable because theyare based on different item parameters.

Ability Estimates For those examinees not included in thecalibration sample, four different estimates of ability werecomputed. For each examinee a 1PL and 3PL ability estimatewas obtained from simulated tailored test. In addition,ability estimates for each examinee for both models wereobtained from LOGIST using the item parameter estimates andthe examinee responses from the 40 item paper and penciltest. This made possible not only a comparison of the twotailored testing procedures, but also a comparison of thetailored testing procedures with the paper and pencil tests.

I Table 2 summarizes the distributions of the abilityestimates obtained for both models from the tailored tests

and from the paper and pencil tests. Table 3 shows theintercorrelation matrix for these four sets of abilityestimates. As can be seen from these data, the two sets oftailored test ability estimates were similar, with acorrelation of 0.77. However, there were some differencesin the two distributions.

Figure 2

The Test Information Function for the IPL

Item Parameter Estimates for the Real Data

-4.6 -. 6 4.0 -. 9 i.0 1.6 2.3 .oTHETA

Figure 3

The Test Information Function for the 3PL

Item Parameter Estimates for the Real Data

i6

!.i

, 9-4.8 -3.6 -2.6 -I,' a.s t.i 2.8 2'.6 3.0

.. . - < .. , . ., . . .. -.. . . -- A.' ° '- . . -*_ " - .- - "..' ' _- C '-

4-,1

For instance, the skewness value of -0.97 for 3PL abilityestimate distribution was significantly different from zero(with a sample size of 1000, the standard error for theskewness coefficient is 0.08), while the 1PL abilityestimate distribution was not significantly skewed. Also,the kurtosis value of 1.96 for the 3PL ability estimatedistribution was significant (standard error = 0.16), whilethe kurtosis value of the 1PL ability estimate distributionwas not significant.

Table 2Descriptive Statistics of Ability Parameter

Estimates for the Real Data

Tailored Tests Paper and Pencil Tests

Statistic1PL 3PL 1PL 3PL

Mean 0.15 0.01 0.21 0.11Median 0.14 0.23 0.16 0.25S.D. 1.36 1.40 1.13 1.18Skewness 0.10 -0.97 0.74 -0.35Kurtosis 0.21 1.96 3.48 4.39Low Value -3.65 -4.00 -2.92 -4.00High Value 6.22 6.42 4.00 4.00Mean Test Length 12.84 12.16 40.00 40.00S.D. of Test Length 4.51 4.73 0.00 0.00

Note. For the LOGIST calibrations arbitrary minimums andmaximums of -4.00 and 4.00 were set on the ability

"* estimates. The same limits were placed on the tailoredtests except in those cases where all items were answered

*correctly or all were answered incorrectly.

-m Table 3d Intercorrelation Matrix for Ability Parameter

Estimates for the Real Data-p

1A10ility Tailored Tests Paper and Pencil Tests

Estimate 1PL 3PL 1PL 3PL

Tailored 1PL 1.00 0.77 0.89 0.873PL 1.00 0.81 0.86

Paper/Pencil IPL 1.00 0.953PL 1.00

4

-11- .

The lPL and 3PL ability estimates from the paper andpencil test had a correlation of 0.95. Both distributionswere leptokurtic (kurtosis = 3.48 for the lPL estimates,

* 4.39 for the 3PL estimates), and the two distributions hadsimilar means and standard deviations. The only realdifference between these two distributions was that the 3PLdistribution was significantly negatively skewed (skewness =-0.35), while the IPL distribution was significantlypositively skewed (skewness = 0.74).

The two sets of tailored test ability estimates werefairly similar to the paper and pencil test abilityestimates. The two sets of IPL estimates had a correlationof 0.89, and the two sets of 3PL estimates had a correlationof 0.86. A comparison of these two correlations viaFisher's r to z transformation yields a z = 2.20, p < .05,indicating that the IPL correlation was significantly higherthan the 3PL correlation. Interestingly, the 3PL tailoredtest ability estimates had a correlation with the 3PL paperand pencil test estimates which was not significantlydifferent from the correlation between the IPL tailored testability estimates and the 3PL paper and pencil test abilityestimates (r = 0.86 for the 3PL estimates, 0.87 for the IPLestimates). The IPL tailored test ability estimates didhave a significantly higher correlation with the 1PL paperand pencil test estimates than did the 3PL tailored testability estimates (r = 0.89 versus r = 0.81).

Average Test Length The average test length for the 1PLtailored tests was 12.8 items, while the average 3PLtailored test was 12.2 items long. This difference is oflittle or no practical importance, except as an indicationthat the attempt to produce tests of equal length for thetwo models was successful. Of some importance is thefinding that the IPL tailored tests required approximatelyone half of the CPU time required by the 3PL procedures. Ofcourse, if this difference had no signicant impact onresponse time, then it also is of no practical significance.

Nonconvergence For the 1PL procedure there was nononconvergence. For the 3PL procedure, however, there was a4.9% nonconvergence rate. Examinees for whom there wasnonconvergence were assigned an ability estimate of 4.0 or-4.0. Of those cases where there was nonconvergence, 96%were at the low end of ability. This is consistent with thefinding that the 3PL test information curve was negativelyskewed and shifted toward the positive end. Nonconvergencehere means that the tailored testing procedure was not ableto compute an ability estimate for an examinee. This could

-12-

happen because the examinee answered all the itemscorrectly, or all the items incorrectly. It could alsohappen if the examinee's ability estimate drifted out of therange for which there were appropriate items before both anincorrect and a correct response were obtained. In such acase, the test would be terminated at 20 items, or when botha correct and an incorrect answer were obtained.

Simulation Data Analyses

Item Pool Calibration The first step in the simulationdata stage of this study was the generation of data to fitthe 3PL model. The true item parameters used for these datawere the 3PL item parameter estimates obtained for the realdata used in the first part of the study. Data weregenerated for 3000 cases, using true ability parametersrandomly selected from the standard normal distribution.Once these data were generated, the items were calibratedfor both the 1PL and 3PL models using the first 2000 cases.The distributions of the obtained item parameter estimates

, are shown in Figure 4. These distributions are summarizedby the statistics shown in Table 4.

Table 4Descriptive Statistics of Item Parameter

Estimates for the Simulation Data

1PL 3PL

Statisticb a b c

Mean 0.00 1.04 0.41 0.17Median 0.16 0.96 0.30 0.14S.D. 0.90 0.34 1.12 0.08Skewness -0.31 0.13 0.09 0.94Kurtosis 0.38 0.11 1.86 1.10Low Value -2.20 0.28 -2.27 0.06High Value 2.00 1.77 3.77 0.40

With few exceptions, these distributions are very muchlike the distributions of the item parameter estimatesobtained for the real data. The only real differences werein the skewness of the 3PL model a-values, which went fromslightly positively skewed to not significantly skewed, andthe kurtosis of the b-values for the 3PL model, which had anincreased kurtosis for the simulation data.

hi..

'1

Figure 4

The 1PL and 3PL Item Parameter EstimateFrequency Distributions for the Simulation Data

3PL A

0.0 o.s 1.0 1.5 2.0 2.S 3.0

_ 3PL B

4 4

C3 L

I IPL C "

1PLB8

i!1-4.5 -3.0 -1.S 0.0 1.5 3.0 .A

4 ~ . ,

V. 4-

One other important difference that was found was that forthe 1PL calibration the items were assigned in a-value of0.60. Since this was higher than the value for the realdata (0.56), it was expected that the test information curvefor the 1PL model would be higher for the simulation datathan for the real data. It was unclear what effect thiswould have on the simulated 1PL tailored tests, except thatit would probably increase the average test length.

Table 5 shows the intercorrelation matrix for the trueand estimated item parameters for the simulation data. Ascan be seen, the 3PL estimates were quite similar to thetrue parameters. The correlations of the true and estimated3PL item parameters were 0.89 for the a-values, 0.99 for theb-values, and 0.92 for the c-values. The correlation of theIPL b-values with the true b-values was 0.88, and thecorrelation of the 1PL and 3PL b-value estimates was 0.88.

Table 5Intecorrelation Matrix for the True and Estimated

Item Parameters for the Simulation Data

Item True 1PL Estimates 3PL Estimates

Parameter a b c b a b c

True a 1.00 0.25 0.10 0.45 0.89 0.21 -0.09b 1.00 0.40 0.88 0.27 0.99 0.29c 1.00 0.11 0.19 0.34 0.92

1PL b 1.00 0.41 0.88 -0.043PL a 1.00 0.23 0.08

b 1.00 0.26c 1.00

Figures 5 and 6 show the test information curves for the1PL and 3PL item parameter estimates, respectively. As wasthe case with the real data, the 3PL information curve isshifted toward the positive end of the ability scale. It iscentered around .8. The 1PL curve, on the other hand, iscentered around 0.0. The 1PL pool once again appears to beappropriate for a wider range of ability than the 3PL poolis, especially at the lower end of the ability scale. Aswas predicted from the item calibration results, the 1PLtest information curve was higher for the simulation datathan for the real data. An unexpected result was that the3PL test information curve was also higher for thesimulation data than for the real data. This was probably

" .'4 , , <,v 4 ,.,, . v . ., % .. ,. - - ... ',, , '" " ' '" "-""'"'", , , . , , -.. '.--...- , ,...i[. ... ,,.; .-....- .... v

Figure 5

The Test Information Function for the 1PL

Item Parameter Estimates for the Simulation Data

0 - . . . . .

C!

40 30 . .0 .0 .0 .0 4.0THCTR

-16-

a result of the fact that the simulation data were generatedto fit the 3PL model.

Ability Estimates Four sets of ability estimates wereonce again computed for the 1000 examinees not included inthe calibration sample. For each simulated examinee IPL and3PL ability estimates were obtained from the simulatedtailored tests -, we~l as from LOGIST runs on the simulated40 item fixed ler.ith test uE-ng the item parameter estimatesfrom the calibration of the simulation data. Thus, all thecomparisons made with the real data results could be madewith the simulation data results. Because these weresimulation data and the true ability parameters were known,the ability estimates obtained for these data could also becompared to the true abilities.

The statistics shown in Table 6 summarize the trueability parameter distribution, as well as all of theability estimate distributions obtained using the simulationdata. Table 7 shows the intercorrelation matrix for thetrue and estimated abilities for the simulation data. Thepatterns appearing in these data are much like those foundfor the real data. For these data the correlations are allhigher than for the real data, however, with the exceptionof the correlation between the IPL and 3PL (simulated) paperand pencil test ability estimates, which was lower for thesimulation data (0.928 versus 0.946 for the real data). The1PL tailored test ability estimates had a correlation of0.931 with the 1PL simulated paper and pencil testestimates, which was significantly higher than the-correlation of 0.826 obtained between the 3PL tailored testestimates and the IPL paper and pencil test estimates ( z =10.954, p < .01). The 1PL and 3PL tailored test estimateshad correlations of 0.920 and 0.854, respectively, with the3PL paper and pencil test estimates. The difference betweenthese two correlations is significant ( z = 7.113, p < .01),indicating that the 1PL correlation was significantlygreater than the 3PL correlation.

The inclusion of the true ability parameters in theanalyses of the simulation data resulted in a veryinteresting finding. While the IPL and 3PL paper and penciltest estimates had correlations with the true parametersthat were not significantly different (0.894 for the 3PLestimates, 0.900 for the IPL estimates), the correlation ofthe IPL tailored test ability estimates with the trueabilities was significantly higher than the correlation ofthe 3PL tailored tests ability estimates with the trueabilities (r = .883 for the IPL estimates, 0.816 for the 3PLestimates; z = 5.452, p < .01). This was rather surprising

I', V , , : " ' . .••, . - " • •, - . . . .

-17-

since the simulation data were generated to fit the 3PLmodel. Just as surprising was the finding that the 1PLtailored test ability estimates had a correlation with thetrue abilities that was not significantly less than thecorrelations between the true abilities and the paper andpencil test estimates, despite the fact that the maximumlength of the tailored tests was only half the length of thepaper and pencil tests.

' Table 6

Descriptive Statistics of True and Estimated Abilitiesfor the Simulation Data

Tailored Tests Paper and Pencil Tests

Statistic True1PL 3PL 1PL 3PL

Mean -0.01 -0.08 -0.25 0.02 -0.09Median 0.00 -0.07 0.00 -0.10 0.03S.D. 1.04 1.30 1.48 1.11 1.22Skewness -0.01 0.32 -0.58 1.11 -0.24Kurtosis 0.14 0.86 1.52 4.27 4.04Low Value -3.82 -3.61 -5.58 -2.47 -4.00High Value 3.74 6.22 6.42 4.00 4.00Mean Test

Length 17.90 13.51 40.00 40.00S.D. of Test

Length 4.05 5.77 0.00 0.00

Note. For the LOGIST calibrations arbitrary minimums andmaximums of -4.00 and 4.00, respectively, were set on theability estimates. The same limits were placed on thetailored tests except in those cases where all items wereanswered correctly or all were answered incorrectly.

Table 7Intercorrelation Matrix for True and Estimated Abilities

for the Simulation Data

Ability Tailored Tests Paper and Pencil Tests

TrueEstimate IPL 3PL 1PL 3PL

True 1.00 0.88 0.82 0.90 0.89Tailored IPL 1.00 0.81 0.93 0.92

3PL 1.00 0.83 0.85P&P l 1.00 0.93

3PL 1.00

,4

..4,<< '.v '' "" " "''"""" "" """" ''' "" " " "' "" "' "

-18-

Average Test Length The avera-e test length of the 3PLtailored tests for the simulation data was 13.5 items. Theaverage lPL tailored test was 17.9 items long. Both ofIthese averages were greater for the simulation data than forthe real data as was predicted from the results of the testinformation curve analyses. The average 3PL test increasedby 1.3 while the average IPL test increased by 5.1. Theincreased length of the IPL tests for the simulation datacould at least partially explain why the IPL tailored testestimates had higher correlations with the true abilitiesand the paper and pencil test estimates than the 3PLtailored test estimates did. Despite the longer averagelength of the IPL tailored test, it should be pointed outthat the 3PL procedure required half again as much CPU timeas the IPL procedure.

Nonconvergence The IPL procedure had a .3% nonconvergencerate, while the 3PL procedure had a 5.9% nonconvergencerate. For the 1PL procedure all of the nonconvergence cases(three of them) were at the positive end of the abilityscale. For the 3PL procedure 90% of the nonconvergencecases were at the low end of the ability scale. As was thecase with the real data, examinees for whom there wasnonconvergence were assigned an ability estimate of 4.0 or-4.0.

Discussion

In recent years a number of studies reported in theliterature have addressed the issue of whether the 1PL modelor the 3PL model should be used in various tailored testingapplications. In a tailored achievement testingapplication, the application of interest here, the researchhas tended to favor the 3PL model. Because of theinconclusiveness of these studies for applications involvingsmall item pools, and because the 3PL model tends to be moreexpensive to use, this study was conducted to determine, fora specific application, whether there is sufficientadvantage to using the 3PL model to warrant the extra

5.. expense. The results of this study will now be discussed,and afterwards some conclusions regarding which model should

"' be used for this application will be presented. First,however, a discussion of the specific application which isof interest in this study will be presented.

The Application

The specific application of interest here has severalcharacteristics which require special consideration. Thetype of application of concern is an achievement testing

* * %?,.- -.%~ ~s .Sp' --. -_ ..V '' 2 .. c'

*-19-

application. Achievement testing must be considered in adifferent light than ability testing because it is learning

* rather than ability that is being measured. While abilitytests generally have learning components, they areconstructed to measure a single trait, and as such areusually reasonably unidimensional. Achievement tests, onthe other hand, are not specifically directed at a singletrait. Moreover, achievement tests often are designed to

*. measure learning in a number of content areas. Therefore,achievement tests typically are not unidimensional, and areoften highly multidimensional. The multidimensionality ofachievement tests causes problems for IRT, since most IRTmodels assume unidimensionality.

One way to deal with the dimensionality problem whenmeasuring achievement via IRT is treat the different contentareas separately. Individual content areas typically arenot unidimensional, but they at least afford a closerapproximation to unidimensionalty than do multi-content areatests. Treating content areas separately presents a newproblem for tailored testing. A single content area of atest may not include very many items. Tailored testingprocedures work best when the item pool has a relativelylarge number of items, with difficulties spread uniformlyover the ability range (Urry, 1977). Building an item poolto meet those specifications, but using only items from asingle contant area might be difficult, and certainly wouldbe time-consuming. It seems likely, then, that at least inthe early stages of a tailored achievement testing programthat treats content areas separately the item pools will besmall.

There are at least two other ways to deal with themultidimensionality of achievement tests in a tailoredtesting application, but at this point neither way ispracticable. One way would be to sort the test items intounidimensional subsets, and treat these subsets separately.However, thus far there are no satisfactory procedures forsorting items into unidimensional subsets when the items aredichotomously scored, which achievement test items typicallyare (Reckase, 1981). Even if sorting could be done, theproblem of insufficient items in the pool would still bepresent.

The other way of dealing with the multidimensionalityproblem is by using a multidimensional model.Unfortunately, no one has yet developed tailored testingprocedures for a multidimensional model. Therefore, thisstudy took the approach of using a unidimensional model withindividual content areas. The content area used was the

V,

-20-

math subtest of the ACT Assessment Program. Using theseitems, a pool of 40 items was constructed. Using this 40item pool, a comparison of the lPL and 3PL models wasconducted. The results of that comparison will now bediscussed, beginning with the real data part of the study.

Real Data Analyses

Item Pool Calibration Probably the most significantresult from the item calibrations was the finding that the3PL item parameter estimates yielded a test informationcurve that was negatively skewed and centered around a pointon the positive end of the ability scale, while the IPL itemparameter estimates yielded a test information curve thatwas symmetric and centered around zero. From these resultsit would be expected that the 3PL tailored tests would tendto terminate prior to convergence for examinees with abilityon the lower end of the scale. Such a tendency would not beexpected for the IPL tailored tests.

Ability Estimates The most important finding from theanalyses performed on the ability estimates obtained for thereal data was that the IPL model performed as well as the3PL model without requiring any additional items. Thecorrelation between the IPL and 3PL tailored testing ability

C, estimates was fairly high (0.772), and the lPL tailored testestimates were just as highly correlated with the paper andpencil test estimates as were the 3PL tailored testestimates. From these data it appears that there is noadvantage to be gained from using the more complex (andexpensive) 3PL model.

Average Test Length For the real data tailored testsimulations, the average test length for the IPL and 3PLtests were about the same. This is as it should be, sincethe information cutoff values for the two procedures wereselected to produce tests of equal length.

Nonconvergence There were no cases of nonconvergence forthe lPL tailored test procedure. For the 3PL procedurethere was a 4.9% nonconvergence rate. Of those cases wherethere was nonconvergence, 96% involved examinees at the lowend of the ability range. This is consistent with thefinding that the 3PL test information curve for the itempool was negatively skewed. Clearly nonconvergence is moreof a problem in this case for the 3PL procedure than for thelPL procedure.

I'%

• -4..., , . .. , . , . . . . ." " ' " " . -. ' ' " "- - ." ' " " - ." . . - . - ' . -

-21-

Simulation Data Analyses

Item Pool Calibration What turned out to be one of themost important results of the item calibrations was that forthe IPL calibration LOGIST assigned to the items a common a-value which was higher than that assigned to the items usingthe real data. This resulted in higher test information forthe 1PL model across the ability range. As a result ofthis, the information cutoff for the IPL procedure wasinappropriately low, which resulted in the tests beinglonger than expected. The test information curve for the3PL model was also somewhat higher than for the real data,except at the extremes. This would also be expected toincrease the average test length of the 3PL tests, but notas much as for the IPL tests. The 3PL curve was negativelyskewed, as was the case with the real data, which shouldhave once again resulted in some nonconvergence cases at thelow end of the ability scale.

Average Test Len th As was expected, the average testlength increased for both procedures. The 3PL average test"-- ]ngth increased by a little over one item, while theaverage test length for the IPL procedure increased by aboutfive items. There is no reason to assume that the qualityof the 1PL ability estimates would have dramaticallydecreased had the IPL tests been shortened by several items,although it would probably have been lower.

Nonconvergence For the simulation data the 3PLnonconvergence rate increased to 5.9%, while the IPLprocedure had a .3% nonconvergence rate. Once again,nonconvergence is clearly a more serious problem for the 3PLprocedure than for the IPL procedure. As was the case forthe real data, the bulk of the nonconvergence cases for the3PL procedure (90%) were at the low end of ability. This isconsistent with the results of the test information curveanalyses for the simulation data item pools.

Summary and Conclusions

A study was conducted to compare the 1PL and 3PL modelsin tailored achievement testing application. Both real andsimulation data were employed. For the real data, the IPLprocedure was found to yield ability estimates thatcorrelated with paper and pencil test estimates as highly asdid the 3PL tailored test ability estimates. The IPL testswere of about the same average length as were the 3PL tests.For the simulation data, an inappropriately low informationcutoff was used for the IPL procedure, and as a result ofthe IPL tests were on the average four to five items longer

-22-

than the 3PL tests. The lPL ability estimates were found tobe significantly more highly correlated with paper andpencil test estimates than were the 3PL estimates. It wasunclear what the results would have been had the IPL testsbeen terminated earlier.

The 1PL model is a more appealing model than the 3PL*.-. model, since it is simpler to work with, requires smaller

sample sizes, and is overall much less expensive to use thanthe 3PL model. The results of this study indicate that forthis type of high quality, small item pool, there is nojustification for the added expense and complexity of the3PL model. For this application, the 1PL model was found tobe the model of choice.

V4.

f-..-.

-23-

REFERENCES

Birnbaum, A. Some latent trait models and their use ininferring an examinee's ability. In F. M. Lord and M.R. Novick, Statistical theories of mental test scores.Reading, MA: Addison-Wesley, 1968.

* Koch, W. R. and Reckase, M. D. A live tailored testingcomparison study of the one- and three-parameterlogistic models (Research Report 78-1). Columbia, MO:

-i University of Missouri, Department of EducationalPsychology, June 1978.

* Koch, W. R. and Reckase, M. D. Problems in application oflatent trait models to tailored testing (ResearchReport 79-1). Columbia, MO: University of Missouri,Department.of Educational Psychology, September 1979.

McKinley, R. L. and Reckase, M. D. A successful applicationof latent trait theory to tailored achievement testing(Research Report 80-1). Columbia, MO: University ofMissouri, Department of Educational Psychology,February 1980.

Reckase, M. D. The formation of homogeneous item sets whenguessing is a factor in item responses (Research Report81-5). Columbia, MO: University of Missouri,Department of Educational Pyschology, August 1981.

The American College Testing Program. The ACT Assessment,Form 23B. Iowa City, IA: The American College TestingProgram, 1982.

Urry, V. W. A Monte Carlo investigation of logist mentaltest models. (Doctoral dissertation, PurdueUniversity, 1970). Dissertation AbstractsInternational, 1971, 31, 6319B. (University MicrofilmsNo. 71-9475)

Urry, V. W. Tailored testing: A successful application oflatent trait theory. Journal of EducationalMeasurement, 1977, 14, 181-196.

Wingersky, M. S., Barton, M. A., and Lord, F. M. LOGISTUser's Guide. Princeton, NJ: Educational TestingSer ces, February 1982.

-24-

Wright, B. D. and Panchapakesan, N. A procedure for sample-free item analysis. Educational and Psychological

A Measurement, 1969, 29, 23-48.

! ..

* .

..

,

4. . .., *55.

4 ~~ ~ Or To. nif f1~m I ~ rlivy Firsacnqi MW Ctntdr Nivy P,!rsonnel. 3t~) Center

%in D1aio. C% 912 %in Dt.!go. Ck 92152

Ior. %rthtur 3 nchr ich I 'like Otirayar'AmvLroam nt it i Qr-ss C' o4r I-% Co rInitructionil, P-ogrva ODvdlopm-nct

4wiv, ii s'tc-t M * -i [c Insiute 8iitLdtnig IC)

G ~-ac% Li.ws orr* EL 600%qI r. 4I!rvt S. siktor

~4ivy P.ors.-a-net R%) lent-1t I tOr. tLchiar.1 K~stnrSin O1C40. CA ')zIsz 0f lrircm t of Isnitr-tcva scL.!nc'!q

I L~~si '~c~mtqc intar--y. f~k 919(1fftica oft 4wivl R-se archSincti )fic.. Lonitrn I flM. P~r FfrgnRfC.

I~~~ ~ %~r I I~'t It I)

ply 'yt ).v I '-. Wl'y 9:' -1j g,4 -1,1-tmroiv~t f~it-q~em Slvy P.'rqonn--t R~f) '7.ntter

WJIRLSi t.v.C125

I fl*. Jtrai rti

I o~r. Rocb-!rt ISr,!iu Cro)IL I's

,i~vra ib ')1) [PrF4 4i ay P 'rin:ria WR %. n 7.mo

ort. anti, Ff. 12-11 1

I o~r. qoh'rt Carroll I v P.%rn 4 ) 'rantor1J1V' in 11fl.v.C. !

Jithnsttfra . nr' !11701 Dlr. 'J)rirnn 1. '(.rr

I ':ht.*f .)f '4ut' 0ii Iwt''itton ,nt rratrninj ('al.'f of *;iv t T''ohLitl r ,itnlnri.* 1.1 vi-an 'If f tc. %(, t tr ;rttlwi !1 **titti (15)

1 r Forr: 'ta iin Riirr-t Lt.hor t ty '411 In v t'r (Ii , r; mv&'

')p.r ationai rritnnqa flivt-.111'4M1L! Vi; %F4. % I iZZ's 1)r*!-. Lntarl < ri-A.-r

Ilivy I',rrind RlY) '.

I O~r. 'S rtmey *Yatly.'r a l.~*' ?

)f ftl ( if I IVivi T-chioilo.ry

;91) 1 Ortc c *. ). 14tiltm I.. AI iloy '!

!JaiIt r t at I'n

I A ik- '.'arrm P- lni a-oli. FL 1STA

S )ff i.:. if *I av% R t i a.- irc it

('ot" 211 'Jtvy P.-r-vniA R4) ':.nr,,r

* ir1liaC!'n. V% 2121U r S in . (A 92152~

I ior. nlzr, I') ,iq I Ur WAtI. tlm at' ~q~

tIFC p,.r -0,1

P*n4it I.F.S n n -to . !i

Wiivy Nivy

I Bill Norrdbrock I Mr. Rrnvi Sympnmni1032 Fiirl~wn Ave. N-ivy Pv'rsrinn'l R&D1 :nte'rLtbirtyvilln, IL 41149~ S-in ni.?go* CA 91151

1 Libriry. C-31-- P2()IL I Ir. Fr-ink Victtv''4vy Pqrs,nnel RVII ',-ntrr N ivy P.-rsoni-.1 R%T) 1-nter

S-in 010go. CAt 92152 Smr Wf..tn. CA 9'.152

1 Terhnicil Director I Ir. E itr-1 W-svmiiiI4ivy I'!rinnn-'l RV) -ete orric'- ei( Nwvai K-si :r-h (' 4111;% SP)

sin, nistgro. r.% 9-lis 41 ) ?4'orth ().alniy Stroo.t

6 Persinnt-t Ii Trutninaw. R .. girrIh Group~ro-le '.41?r I Or. R''niI Id .W-tzimnOff lc. 'f Nival R-s~irvhI 4tval roqtstrifitt S-)--lArliIton. VN 22217 1) pisrtnt on f '4-1ainsritv'%

SC' I flfles%

I So.!ci it 44uw. for E-fiacition .inif Ahnt.r.y. (% Q19V.)* rrailnin% VP-)lF.)

Rm. 2705 Arlinton Ain-'s 1 0'. ODai'l-iq 4-zt

W ihin-mtou. Wl: 201701 Cot.o I?Naivy P.-ru'i'n -1 IRaF) rntt-r

I LT Frinlc C. P.-tho. PI'V;, INN (Pli.fl) ';in DI1 -iPn. C% Oq215!rNer (N-M3) n.~ar~~ va~v

-NAS R jRjJF %KF

P'*n-&ican- Ff. 11.504~ 4%VY PFR';')WI, 1) '4&TPEt

I Dr. R-rn-irv Rintagn-I (01C.)

Nivy P.'rqeonn,- RhDr Cc-ntcr I 4r 1Ihn 11. U)If"evin Dtnflo. ('A 92152 N ivy P,-rq.oori-! RY) >'nLrio

';in Db~.CA 92?152I Dr. C-ir RoRis

CNIKT-PD71) I Dr. Urn lacv'! W,i1fock. IIIBuildinj 90 N-ivy P-rsnnl R!.I) Centa'r

Creit Likes NrC. IL 6914 1 S-in D1..ign. C% 92152

I Dr. Rob-'rt G. Smtth- Office of Chief of Niv.Il otveraionq

OP-91711meowWauhtnqton. D-1 2015')

I Dr. Aifrel F. Swvilr, DirectorTrtilntng knrnlysis & Fvigriton rCroupD'-pt. of th! NivyOlni o, FL 32411

I Dr. Rirhir.i Seiranq-n

Nm -ivy u Pesi-i R)C t-

I Dr. Fre~i--rtck Steinhi-IserC413 - OP115N-ivy Ann.!EArlington. VN 2037n

.1 Mirin*t Corps Army

I H. Wt II tit .* re !niv 1 Technir~il Dtroztor

EIiii:ton 44vi sor (Efl) 1 1. S. Ariv R'q%-.irch Incttjtat for th"!

Oliainttreo, V% 221 I's 5311 Fls,-n~v)ir A,.'nir-fklox,.nrii. Vk '2 V13

1 Dtr'crcor. ffftc, of lvinp(i4.r titil Iiin

11). 'lirin,- f:Yrpq ('4P11) I Mr. Jvn'un !Iik.'r

9qv1 tIrn. V%.'II' V4 Fist nvir jN. nw-A~oic nIri m . V% ?"II

I ' * Iatr~r. 1.S. *liriiu * (ovpt( 'I I~ I Dr. K'-nr Fiw

W %i ~I fl',tfi . D-2 11 % r-nv R. - -;, i r.-h I nt ( t u t

I qp'clil Nqsistnr for 1,trin Al-cintrli *P V') "Il1C.,rnq 41tt-r4

co t.o 10 )1 1 Or. $1-itrlc'. 1. FirrOffte.. -)f Nii *s" t-ir U. S. Armuy R-s-ir-h Inqtriit,-81~) 4. 9-sinhy St.i'IE Kq'-n~t,.,-r Avvni-

I n. A.L. SIAP\<)S'(Y I Dr. Myron FlqrhI%iiFrI FTC AI)VI S )'t(Y 4O1-) U. S. Army R..q.. arch Int itt.* for thw

* W). U.S. 'IvtIN -)-w; Iriv 1:1 -tn R.Thivior,l Srl,-ne-'q

WS1I IN:.r, n:2 2 1~' S'f41 511tq-'n'inaur A".wnu

1 9 ijor Fr ink Y'ohinnin, u';921I-aacqteirtrs. 9irln- Ci)rpq I Dr. Milton S. Kit?.

Wiqtedniton. v' 21'13) U.S. Arliy R-a.* rcia (nqtt'-Ffl)1El qnbn*4r Avnii,

I nr. lirold F. O'N.-il. Jr.

Dir.!ctor* Trilning Ros-irrch L-ehNrmv R'-m.irrh fnqtlttt

-~ AI'Eainili. V% 22.111

1 C)-viaiinir. U.S. Army t l-s,rcb Tnqtfttt"for th- R~ voi .51a clonros

ATTN-. PFql-RR (Dr. hilith n)rictnti)5')')I Flq('nhiv'r Avi'nu'

VAI"Ka.-nril%, V% 2r)III

1Jos(-Ph Psotki. Ph.D.ATTN: PP.47-IrAriy R.R-ir Inqt itt5011I Elq-'n~iiu 'r A-,,,,

A1,!xcin-IrI-k. V% 21113

- Wh; C W.Zw. i L ? * * J W. . ' X . . % . ** ~ - . - - '-

Army itir Force

* I Mr. Rnb.'rt Ross 1 41 r Forcl- Htrin R'soirrtos LaibU.S. Army Reisearch IntIstiite for th- AFIRL/4Pfl%2ct it ant R-'hitviorril So-esnr's lrriolk APR. TX 71115

4l~Eanr~,Vk 22111 1 T"ch-itc-ii Da-etnts Cnt,,r4 %l r Fore'. tIhstn R-so)tsrr-s Libor~itorv

I Dr. Robert Sisvor wpAF~i. (Vi 45'1m11. S. Army v " r) In."Ilrlit f'sr Lb.R-h'vtnrail -in Sewit. l* I U.q. Ai r For'..- 'ff ir.- of Silntlfi'-

NI..witilrla, VA 22111 Li fe n .n- i r.-etor itit NL'milln-w Air For'- Rt'a'*

I Dr. lov*. W4t.liqhilncrrn. Or~ 211112'kriy R*'s.-mr'h lnqtttu-. feir Ow.01-hivicr-rij ivn-I i SI':~~ I Air Ifniv-rstty Libriry

S I D~r. lif1.1. Winq, I Or. Fi r I A. AI lIisIArmy Ri's - ire-h tunnt I ttt- 11). AFIqIR. (1:%*-)S)11l Ricis'nh-wr Av.-. Ilroalkq %F3. TIC 74215~

1 %fr. Riy.tmail F. Cirist-iII nr. Rob.-rt 141 lt .r AFIIRI. I'')F

Army R,!s-i rciiln Inflcts Brookq AP'i. TX( 74115

%1(-x~inlri.-i. VA211Dr. Alfril R. FrogIy

I 1Dr. C-n'rie ve II id ;i-Program linav'rLife Sclenco-s rDlr(r-torawtpAF')SRAroiitnq %F5I. D^' 20l312

1 Dr. T. M. LringridgvAFIIRL /nTE

'U. Wtilta'u APR, A" 9522'.

I Mr. Rinnoph Park

AFlIltL/M') V

Brooks AFS. TX 7R215

I Dr. Ro~i.r P'nnolIAir Fore 'Its-n~in R -sniircoq Labhor atoryLiwry V4I. C) 81)211

I Dr. "tilcnlm RoekFIIRL/MPBrooks APR, TX 78235

M7 . j .. .. . . '..-

ktr Fnrce D--pirtmqnt of Dl'fun-i

1 3711) TCmrw/rG'IR 12 0--fnnv T--?hilrcil Inforvuition Cntrr

Shv'npiri *AFR, TX 76111 Al.!-xanvri-i, V,% 22114kttn: Tr,

I Lt. C--i Jimu!s K.. Witson111) US %F/'PXv~i I Dr. rrilp, 1. Flt'-isTlv- P.'nt-isgo ki-tV'-i R.s-i Pro10'ctS i~j~r

Ar1inton. V%~ 71109I *1 io#r Win'n W *I-'i

*AFK,.i: I Jirry I.-ni

1%, tn: 9'il-Y~-p

D Ir. .l~.p~Y ti eWu Fort S't..ridiun. 11, (oV3)7AFOIR/Lit rL-)wry AkFR, r') 'i'll) 1 '11 11t iry RP.:iq~nt f or Tr-itn i n in I

V ~)f ftc C4of th * '7'Inor S.'er'-t ry -if I) *fnsfor R -i.'rch ,iI--i,

I fOr. Wiyn.- S..I-ainflfflc- of tls.' AMqisr irit 4q,-rrot ory

* ~~ of f) .f.-nq.' ('1'RA ft L.)

Wa1h11w'groi, D" ?1)1(1l

I 'titor .1ick T-inrp'-

1401 'Ailson Riv-1.* .~ *rlingron. VAN 22209q

INKOZ

Ctvtlt.t~ kqmnees Prtvat.-';S-rtnr

I Dr. Suu.In Chitpi2in I Dr. James AliniL--irntnq ini Davenops'nt Univo'rsity of Florti4iNitto-it I nstitut.? of 9-toicnion jrrqiI. FL 12r%12n') 19th 'Strn W4

Wishiniton. MC 202)3 1 Or. Erlinq B. Aii-rin

I tr. V-!rn W4. Urry Sti rq~ ti.'P.!rsonnOI R%t) -'nter 145~5 ro~i'i-nff ic.' of P.-rsmn.' 4in sq.'u*nr

,shinp~ton. W: 2')'415 1 1 Pqvrh,)Io.#lr-o R-%ic '''trei ntNJIVI- I-Vi' %t tn- LI h.irtin

I Mr. T;,vuiq A. Wa~rm W) r t hS uiirn, II 11)m41). 1q. Ciit G-a urt I n~t ituace Ttirn r kr r 26111P. 1. stah~tition jqNl;19%1)1'viCity. OK 711641 r soR-i

I Dr. Jns--P'i L. Ynitnev, rrctor riw-irlint T.*qtlI,,-'rvi.-M-!,nory &. C'mvnittwi Pro.:,'v'q Prlruc.ton. III 'I','e(Nitiontl Sc-it'n-.' Fnu'-littrinWniqlincwton. W 205YV) 1 nr. Mnit'ii ltiIr.'nh aii

ri Av Iv U I v-rq I trol Aviv. Rl'"'t Aviv $0~97414 r it- I

I Dr. Ri. Dirro'II RikO-i)p rtm-ncu of F Ii,"r 11)Unlve-rsltV ift Citic-11o('iI":a',o, IL f')17

I nr. Roh-rt 8r-ni nkm-iarea relig..- T*'attntv Pros'.rvaqP. 0. Rox 16MI'

I Dr. Ern-st Ri. Ci-Iraitte3')? stok-l.yIIntv,-rqltv -if T,.nn,,sqn.-oKnnxvtllr, Tif 17416

I Dr. Jorhn R. Cormrll409 911tritt RI.

IleI Or. N-irmin iliffn-pL. of PqV-%-ilO'yUniv. of go. Cailfornim

Unlvorsity Pirk

Li' ngSq C ))

Private S-ctor Prit.l %--ctor

I Dr. Ilins Crombig I rnr. D..,cfr'r F1-tch-r*E'Iuc-ittrm R-s.i-,rchi :-,nt. r WICT R'-eirch Institat

Univertty of Liy~vo 1975 S. Stin St.Ii) 'rlii~v',1 Iren (ru U 22113

2111# E4 L'qy.i'nh flI NMLANT; I Dr. Jintr:.- Gifford

Ij'ni vrst ty of isih,'tI Or. Dmctpraddi D)i vpI ScdI'ol of Fliat-ition

* Syvr.iriq, UqI vers Ily VNirsr. 'N 11012D'pirt-n'nt of Psychti-n-'.y

*~ri-i- 1 112101 1 nr. Rorl GI i-'r1 am nn" 'R's",rh %, O- velopi-nl rC"-

I Dr. Fritz. Dr.tqqno4 Juilvorslty of I'ittchruiO) p artm-nt of Pqy,-haloa'y I10 ') Il ir.. ';tV't

Unulv.rqltv of 11inulq P% 151fII)') pF)TI 9. lanl-I St.Chirnipiiqn. I1. 618~?-) I )r. Ilort Grae'n,

1ohnq llnfik ins tin Iv.-rq t yI Dr. Saisin Ewhrtq-in 0-p .rtmntn of Pty'-hloW!

* 1AI1R1RSTY IF K%1;v%; fll r'o '1') 71219

l~ir.,(-,. K 66'#51 D~r. Rons If iiibll'oI FRIC FilLLy-Acquil-tclonq Sr~hvioI of Etaaai~on

4511 5aL',y Av'-nu", Unlv,-rslty .Vf M%4q-ichuiqct~q-'p *'chosit. 40 M 2()')14 lv'ir., 1 I (11')

I Dr. I8.nj.-vi-n A. F.itrh ink. Jr. I Or. n-Iwyn 'I-irntsei'17Fnn-Cr.iy S, Alqocltite, Inf-. 1unIverslty of 111Inniq541 SI?'a L~ ~~iin ?4?h F-iur it IonS-ilte 2*15 Ilrbin i. YL (119 )1S-in Nntoruto. TX 7,125

I Dr. P-ml lnIlirtI Dr. L -on irJ Fro1it 677 G Stre.t. #114

Linilqist C.!ntnor for M 'uqrmlint Ctmu1l;1 VIREui, C,% 90(110)4 LJqiv'rqIty af 1041

Iow., City. it% 52242 1 Dr. Llovi timnapirc'vsDiprti'nt of Pqv-ilngy

II Dr. Rich ar-I L. Ft.rqtison Unive.rsity of Illinois

P. (. M-IX 168 Chiinipiin. IL 6192-1low I Cl ty. 14 l24(

I flr. Tick 11 intv'rI Univ. Prof. D~r. G-rhir Flqc~hnr 2112 Cootldp.' St.

Lt~b~as"5/) L-insing. M 49916A 1010) VlenniAUirRI 4 I Dr. 'lIayn'i 'biavnh

'o11.er-" of Elue-ittonh*e1 Profos'bor Drn ide Fitzg',-rilI Unfv--rstty of Siuth rirolini

Uilversicy of N-4 FEngtant Coluuabi. q17 291-19%rmtillN-w Sotith Wuiles 2151

AI-TRAL 14

PrIv~tte S"ctor Privaite S'ictor

Dr. Dauxlas H. Jones I Dr. Frea1'ric 4I. Lnri1

Prorporat ton Prtnrv!ton. NI 03S4u1110 TrifaLir Coirt4

1 nr. T1im-% Ur.lqr~nLi-wrrrncviLle. 41 041149 fl-pirtm-nt of Peythinin

IPrnot"'qo)r John A. K,!it s 4 M I in'Iq W.*A. 6119no irtm,nr. of Pqye!Inio-jy AINTRNLINTh- lint v-rsi ty of N.'v- ust I

*!I.S.4. 21- 1)~ O r. Ciry Mircri

is 'Ir RAL14 'tnp 3I-r

I Dr. Wili tit '(od-i Prlnroton. t.Jj '31'*S1Uiversi ty of T "(:t%-,AjqI n41 .#-iur'--wnr .n-I Rv~iluitloti C',nt'-r I fir. 'ott 'fix.,114aistin. TX 1'infl n-pirtm-ni of P-iyrhlno-,

I nr. jk1en I.-R~igoij Nitro. nim-. 1446S

Univ-rg4Ity of PIttqh-irqh I l)r. Siiiawl T. 4ivai'93' ')'Ifiri Stro'-t Liynt a UIiIversity -if r IrqPittqbtrgI. P% IS2%1 920 *-hrth Mlchiqin Av.-nii

Dr. lllchw., L-vin'D-p.trtm'ir of F.'uritton-a Psych-uin.1y I 4r. Rbihrt Mr'~inl#'y210 !K't-ition 811g. Aia'-ric.in 'otlv'iw.. T.-qtinq ProgriusUntvisrsity .f IlLinol.-i P.O. Rouc 164Ci'aptign. IL 61111 lovi City. 14 S2?4'1

I 7)r. ChirI'-q L,!wfs I nr. Rithir.ii.inFiculttet Sok-t.il W -tennchipp'n 'thin in R t.niir cf!q R(, q i r rh Ort in iz i t tonRi IkiunIvaers It.!t t Cron inig.n 11Y) !4Oirh Wis'iincton

9712rC Croninq.-n4- th'rl-innis I O~r. Rob-rt Plislevy

711 T11trvils Strr-tI D~r. Rnh,-rt Linr Gpnovni* IL 41)I14

CrillogL. of E'IucationUnive~rsity of TiLInnis I nr. Allen 4tinroUJrbini, IL 61411 IIehivilori Torhrtlogy Lihbiratortoq

1945 Fla'ni Ave.. Votirrti FloorI r. Pill ip Livinstori R.-ioiin, lInich. CN 9n277Sy-ttemq in- 4palt-i grienr-q Corpnitto6q1 I K-nt lworth 'kv,,niv I Dr. W. Alan Nic'-win.-hrRivtert-ile. 411 20111'0 Univerqtty of 041niimu

n-pirtruont of Psyrh1innlyI Dr. Robert Leockmin r:Ia,~fty. OK( 73('S9

Cintt!r fo)r Nivit Anslysli2i),) 4cirth Risureigarl St. t Dr. 9'Ivin R. 4ov IckAtexineirti. VA 22111 356 Ltnlquist r nter for ?iq~at.iant

University of to-.iI towi City. IA 5'.242

re.A4

Private! S-ctor Private S'!ctor

I Dr. Ji-nis Olson I Lowjll ';cho-rwlCV\T. Inc. PsYedi1o0FIca1 & ')ainiditive1875 Sotith St'u,.e Srn !t Foti-ttlonsOrtm. uF Avm5 Coleg of Eiiw'itton

U~niversity of lovaI 'Jiyni' 4. P.-tten-± towt (lty. 14 5?21#?

A,!rtc an f'oaarvtL on E'lia-tttoaCFJ) Teostln,, S-rvlre. Sailto 21) DR. R1,3!".Rr .1. FiI),:L0)-- r)-.ipot (r1',i- N4 I NSTRuirmF OYL TF. I41,)1AY GRO11P

I Dr. .J am Aq*. Pautls-i %L9M'M1%ll. VA~ 22114Porr ian i qt at,- Uai veor i tyP.O. II)x 751 IDr. Kizti3 Shig-t.,

* Portli-iii OR 9-71?)7 Untv-rql ty if Toa-kAi0- p artiant of F.itic tio, ml I',yt-Yio)r'1v

IDr. 41ark 1). R-ckis* Kiwatachl S-niat 9141)4

P. 0. 3,)x 164NIowai Cley. IA 52114 I Dr. -wl n Satitrk -v

D'prtm'naof l'qvy-1jaoorVIDr. FaiqR-yanljs Unilva-routty of rr-ntrui Florli.,Unlverqtty of Tox.aq-'l i:is Irtlanin. FL 12)4164irk -t t nsv 1) -p-trtmantP. 0. 8Bx 6qg I Dr. WilIipaa Slmq

* Richaratq',in TX 751)l C,-ntr for Iay-al \,i alvqtq

2)91 Morth 4-tirog irr Stro-"t1 Dr. L awri-n,,' Raiin.r AI'xiniria, V\ ?2111* 4')I )Elm %vt'nu'

T-Aiki Park, '11) 20112 1 Dr. It. Will ac- Sli ikoProgram Mir.rtor

I Dr. J. Ryin '1:tnn-)w-r R'-;' arrh -an I Alviqiiiry SorvireqM.,partmeknt of F.'Iu:ation, S-aithsqntan Inqtttar innIinivs'rstty of Soiuth Carol in a 803 H-)rth Pitt Str.,"tColumbia. SC 2921 1 Al-xanlii, V\ 22114

I PROF. FIPIIK) S,\4Jt!4, I Dr. Roh'rt Stternh'rvD'spr. OF I'Y'ii.O,-y I)-pt. of P-qyf'hIotyvU INIVERSITY O)F rFNmssI:I Yatl, tJn~v(.rnttyKNOXVILLE. T4 17916 ti)IlkW. Yale St-ition

S-w Iive. CT 0,S521I Fraink L. Sch-tiidt

D'p artm 'nt of Py-h1, I Dr. P.'tu-r Stoloff11~149, cr C':ntr'r for Nav.-ai \n i IV. q

roog. iashingtai) Univ.rsitv 211 4orth 111-aairard~r Strot~"Waihtn-,ton. D17 20152 A1"nxanirla. VA 22111

1 Dr. Witter Schii~tdn-r I Dr. Willin- qtot*P-4ych,)Loqy fl~prt,-i-nt Urili~rSit of It1in-fi,;*611 E. Daniel. rlioartmsnr of 9uith2u-tl'q

Chii-iqtn. IL 6182') Urbana. IL 61801

* Privac.' Sttctor Privite Se'ctor

I Dr. Iltrihirin Swi-sinithin I Dr. !Irt-in Witirs* Libor:stary of Psyrhimutrtc 3fl:1 Itu-nRR')

Rvalitid Resn-ich 711) 4-)rth Wishtnixton

Sihool of E-ucatto.- Nl-xinirli, V% 72114University of M-iss~tchijitts r lvl .'~qkuith-rst. m.% nio-ti In.ntt1

I Dr. Kikimi Titstiok-i Univ..rqIty of '11nn-ot i

Cvimp.ste.r R-osIt, 9icat to- R 's-.jrrch Lih 15 l.Vvr Rn) %I

4 I nr. Rin I R. IV~n'I Dr. "I iirtic- Titsw~k i UnIv.,rctty nif S-'if'h.-rn ".1l I fo-nf i

221) PIu- it Ion f1Ihg 1) 'p % rt'tnt o~f P.y-11-11 o'!y1310 S. Sttt St. Lqq Al- -. r:% 9))I

Chvrupittr. IL (hI1920

I Dr. Divid Thir-n St re itkr i-fto 1k tI)'pirtm-'nt of Pqy'-!hiIoly B,),4 1 5') It1Univerty of '(inctis Dl.510) Rnan 2

I Dr. It'b.crt Tsut:,k iwo r Brue-' 1111 II 1 ms

rl'P;.rtwu'nt of Statistics 1)-p-irrm-nt of Eltir-itlnill Pqy-hl1OnlUnilversity of Missouri Unt-lorsity of III Irif~s

Cnoltbi. ') 65201 Urhm-i. IL 618)1

1 nr. .1. Uhlin~r I rt. W.-n-Iy YfenUh i ner Consul tits CTI4/V-r.-iw ',ItI1I

1 Dr. V. It. R. UppalurlUnton i Trbi,]n Corpar.itloiNqizIcir DivisionP.O0. Box Yflak Rtdqp. TM 17R10

I Dr. D~ivid Vilf!Asssua-nt Syqtemri~Crpirittlon2211 Univorsity Av'aits-S'Ittc 310)St. Paut. 41N 55114

I Dr. Hwir-i W sln'rDivision of Pnyzhi~ij1eil StullesF.iucition-il T..stinq Srvlc-Priceon 41(V

I Dr. Mtchiv'l Tl. Willern!pirtus-nt of Eiti-itonil Psyc-hilogy

* Unive~rsity of Wtsronsin--Mtlw-%Auk!c4twi~vu'v0'. W1 51201

AF

xU

I-.- - --n. rr ---- - -- -- - - --