Item Analysis for Key Validation Using MULTILOG

American Board of Internal MedicineItem Response Theory Course

OverviewOverview

• Analysis of multiple choice item responsesAnalysis of multiple choice item responses using the Nominal Response Model (Bock, 1972)1972).– The NRM can be estimated using Multilog.

MultilogMultilog• The Multilog program estimates the parameters g p g p

of several IRT models for polytomous data:– Graded Response Model– Nominal Response ModelNominal Response Model– Multiple Choice Model (Thissen and Steinberg, 1984)

Lik BILOG MG d PARSCALE M ltil• Like BILOG-MG and PARSCALE, Multilog was written specifically for IRT analyses, so estimation is very fast and efficient.

• Much of the syntax for Multilog is similar in syntax to that of BILOG MG and PARSCALEsyntax to that of BILOG-MG and PARSCALE.

Fitting the Nominal Response Model in Multilog

Multilog ExampleMultilog Example

• To demonstrate how to use Multilog toTo demonstrate how to use Multilog to produce estimates of the NRM, we will use data from:data from:– A 20-item test (simulated for demonstration

purposes)purposes).• Each item had four score categories.• Analysis of each item’s distracter options will y p

happen.– 1,000 examinees took the test.

First Step: Data File FormatsFirst Step: Data File Formats• Prior to using Multilog we must set our data into g g

a useable form for the program.

• For our example data, we will the same file used in our BILOG-MG example.

The file had the response option selected by each examinee– The file had the response option selected by each examinee.– In BILOG-MG, we needed a key file to score each examinee’s

repsonse profile.

• Unlike BILOG-MG, an answer key is not needed.– Just the option each examinee selected for each item– Just the option each examinee selected for each item.

Data File FormatData File Format

Running MultilogRunning Multilog• Multilog is a script-based g p

program that features a graphical system to help set up the script filesset up the script files.

• To get started, openTo get started, open Multilog.– Start…All

Programs MultilogPrograms…Multilog 7.03…Multilog 7.03

Creating Multilog ScriptCreating Multilog Script

• To begin, we will run the NRM on theTo begin, we will run the NRM on the example data.

• Click on File…New.– This will create a new * mlg file containing theThis will create a new .mlg file containing the

script that will run the analysis.– Be sure to place the script in a folder that has p p

fewer than 80 characters total (perhaps on the root drive directly).

Multilog ScriptMultilog ScriptNRM example...distractor analysis.

>PROBLEM RANDOM, INDIVIDUAL, NITEMS=15, NGROUPS=1, NEXAMINEES=1000,NCHAR=5,DATA='EXAMPL01.DAT';

>TEST ALL, NOMINAL, NC=(6(0)15), HIGH=(5,5,5,5,5,5,5,5,5,5,5,5,5,5,5);

>SAVE FORMAT;>ESTIMATE NC=2500;>TGROUPS NU=10 QP=(-4.5(1.00)4.5);>END;66123459111111111111111222222222222222333333333333333444444444444444555555555555555666666666666666(5A1,15A1)

Multilog Script AnnotatedMultilog Script, Annotated• The initial portion of the script contains the name p p

of the analysis

NRM example...distractor analysis.

• Unlike BILOG and PARSCALE a comment• Unlike BILOG and PARSCALE, a comment section cannot be included.

• All other command sections must start with a > and be terminated by a ;

PROBLEM Command LinePROBLEM Command Line>PROBLEM RANDOM, INDIVIDUAL, NITEMS=15, NGROUPS=1,

NEXAMINEES 1000 NCHAR 5 DATA 'EXAMPL01 DAT'NEXAMINEES=1000, NCHAR=5,DATA='EXAMPL01.DAT';

• The PROBLEM command supplies the general information used by the Multilog program (like theinformation used by the Multilog program (like the GLOBAL line in BILOG-MG).– RANDOM specifies that both item and examinee parameters are

to be estimated.– INDIVIDUAL specifies that the data are in the form of individual

response vectors (can have aggregated data, too).– NITEMS specifies the number of items.

NGROUPS specifies the number of groups– NGROUPS specifies the number of groups. – NEXAMINEES specifies the number of examinees.– NCHAR specifies the number of characters in the ID string.– DATA provides the name of the data file for the analysis.DATA provides the name of the data file for the analysis.

SAVE Command LineSAVE Command Line>SAVE FORMAT;

• The SAVE command instructs Multilog to save estimated parameters to filesparameters to files.– FORMAT will write the file in a useable format.

ESTIMATE Command LineESTIMATE Command Line>ESTIMATE NC=2500;

• The ESTIMATE command supplies the estimation specifics for the program.

– NC specifies the maximum of EM algorithm iteration cycles.

TGROUPS Command LineTGROUPS Command Line>TGROUPS NU=10 QP=(-4.5(1.00)4.5);

• The TGROUPS command supplies the number and location of the theta quadrature points.

– NU specifies the number of quadrature points.

QP gives the location of the points– QP gives the location of the points.

END Command LineEND Command Line>END;

• The END command tells Multilog that the command part of the syntax has terminated.command part of the syntax has terminated.

• More syntax is needed still:– The mapping of category codes to response

parameters.– The variable format statement.

Response Mapping SyntaxResponse Mapping Syntax6123459123459111111111111111222222222222222333333333333333444444444444444555555555555555666666666666666

• The response mapping provides the key for Multilog toThe response mapping provides the key for Multilog to map the character responses in the data file to those which will be modeled by the NRM.

• The 6 gives the number of response options.The 6 gives the number of response options.• The 123459 gives the actual options. • Each line after that codes each of the 15 items into a

response for eachresponse for each.

Variable Format StatementVariable Format Statement(4A1,1X,15A1)

• The variable format statement is a required statement that lists the way the data are stored in the data file.

• This statement uses FORTRAN-like syntax for reading data from files.

• The first statement, 4A1, lets BILOG know there is a column of data for the examinee id files.

– 4A1 gives the information that the column has a width of four characters.

• The 1X states that there is an empty column between the examinee id and the data.

• The 15A1 states the data are contained in the next 15 columns, each with zero breaks in between.

Running MultilogRunning Multilog• Now that the syntax has been created, the Multilog program must be

runrun.– Multilog runs in single phase.

• To run the analysis:To run the analysis:– Save the input script file first.– Click on Run…Run

Wh t h ld h ft th M ltil l i• What should happen after you run the Multilog analysis.– After a few minutes, you will hopefully see a set of text files pop-up in

the Multilog window saying things completed successfully.– Other times a pop-up box will appear that indicates an error within the p p p pp

program.• Debugging can be difficult in Multilog.

Multilog OutputMultilog Output

• To view the output from the MultilogTo view the output from the Multilog session click on View…and select the only option (named after your syntax file nameoption (named after your syntax file name, but ending with .out).

• An example from one item’s output is h th t lidshown on the next slide.

Multilog Item OutputMultilog Item OutputITEM 12: 6 NOMINAL CATEGORIES, 5 HIGHCATEGORY(K): 1 2 3 4 5 6 Item Parameter A(K) -1.08 -0.04 0.38 0.15 -0.38 0.96C(K) 1.22 0.64 0.45 0.42 -2.55 -0.18

CONTRAST-COEFFICIENTS (STANDARD ERRORS)FOR: A CCONTRAST P(#) COEFF [ DEV ] P(#) COEFF [ DEV ]

Estimates

CONTRAST P(#) COEFF.[ DEV.] P(#) COEFF.[ DEV.]1 111 1.04 (0.15) 116 -0.58 (0.13)2 112 1.46 (0.18) 117 -0.78 (0.15)3 113 1.23 (0.18) 118 -0.81 (0.15)4 114 0.70 (0.61) 119 -3.77 (0.52)5 115 2.04 (0.20) 120 -1.40 (0.19)

@THETA: INFORMATION: (Theta values increase in steps of 0.2)-3.0 - -1.6 0.055 0.068 0.085 0.105 0.130 0.159 0.193 0.231-1.4 - 0.0 0.274 0.319 0.364 0.406 0.442 0.468 0.482 0.4840.2 - 1.6 0.474 0.453 0.425 0.393 0.360 0.328 0.299 0.2721 8 3 0 0 248 0 227 0 208 0 191 0 176 0 162 0 1491.8 - 3.0 0.248 0.227 0.208 0.191 0.176 0.162 0.149

OBSERVED AND EXPECTED COUNTS/PROPORTIONS IN CATEGORY(K): 1 2 3 4 5 6OBS. FREQ. 390 173 161 144 7 125OBS. PROP. 0.3900 0.1730 0.1610 0.1440 0.0070 0.1250OBS. PROP. 0.3900 0.1730 0.1610 0.1440 0.0070 0.1250EXP. PROP. 0.3927 0.1727 0.1597 0.1432 0.0070 0.1246

Viewing Item ParametersViewing Item Parameters

• Multilog makes viewing the item parameters a bitMultilog makes viewing the item parameters a bit easier with the inclusion of IRT Graphics, a package for plotting the estimated IRFs.

• To view some of the item results:– Close the output window (the lower “x” at the top

right).– Go to Run…Plot.– The IRT Graphics program should open.

IRT GraphicsIRT Graphics

• To view the item parameter results, clickTo view the item parameter results, click on the ICC button (at the top right).

• For each item, the ICC is plotted.

IRT Graphics ExampleIRT Graphics ExampleItem Characteristic Curve: 5 N i l R M d l

0.8

1.0 2

Nominal Response Model

0.4

0.6

Prob

abili

ty

0.2

1 34

5 6

Category legends Item: 5

0-3 -2 -1 0 1 2 3

1 3 5 6

Ability

Solid Lines: 1= Black 2= Blue 3= Magenta 4= Green 5= Red

Dotted Lines: 6= Black (High Category: 5)

IRT Graphics InterpretationIRT Graphics InterpretationItem Characteristic Curve: 5 Nominal Response Model

• Consider Item 5H th t

0.6

0.8

1.0 2

bilit

y

•Here, the correct option was Category #4.Category # 4 is the

0

0.2

0.4

1 34

5 6

Prob

ab•Category # 4 is the most likely response option when theta values are high

Category legends Item: 5Solid Lines: 1= Black 2= Blue 3= Magenta 4= Green 5= Red

Dotted Lines: 6= Black (High Category: 5)

0-3 -2 -1 0 1 2 3

Ability

values are high.•Category #2 is more often selected when theta valueswhen theta values are low.

ConclusionConclusion

• This afternoon’s introduction to MultilogThis afternoon s introduction to Multilog scratched the surface of the things the program can accomplishprogram can accomplish.

Th NRM b fit i M ltil• The NRM can be fit using Multilog.– Distracter analyses are one outcome of the

NRMNRM.

NextNext…

• Transforming Cutscores into• Transforming Cutscores into Theta Values.–Standard setting.–Score reporting.