Download - ABCs of IRT
![Page 1: ABCs of IRT](https://reader034.vdocuments.net/reader034/viewer/2022052307/55895212d8b42af80b8b4586/html5/thumbnails/1.jpg)
November 18, 2010
Diane M. Talley, MA
Stephen B. Johnson, PhD
James A. Penny, PhD
![Page 2: ABCs of IRT](https://reader034.vdocuments.net/reader034/viewer/2022052307/55895212d8b42af80b8b4586/html5/thumbnails/2.jpg)
Psychometrics as Science and Art
2010 ICE Educational Conference
![Page 3: ABCs of IRT](https://reader034.vdocuments.net/reader034/viewer/2022052307/55895212d8b42af80b8b4586/html5/thumbnails/3.jpg)
� IRT and Classical
� Concepts of IRT
� A logit
� The abc’s
� Benefits
� Pre-equating
� immediate scoring
� Population invariance
� Assumptions
� Implications
2010 ICE Educational Conference
![Page 4: ABCs of IRT](https://reader034.vdocuments.net/reader034/viewer/2022052307/55895212d8b42af80b8b4586/html5/thumbnails/4.jpg)
The right tools for the job
� Data
� Program
� Tool
2010 ICE Educational Conference
Versus
![Page 5: ABCs of IRT](https://reader034.vdocuments.net/reader034/viewer/2022052307/55895212d8b42af80b8b4586/html5/thumbnails/5.jpg)
Classical versus IRT model
2010 ICE Educational Conference
![Page 6: ABCs of IRT](https://reader034.vdocuments.net/reader034/viewer/2022052307/55895212d8b42af80b8b4586/html5/thumbnails/6.jpg)
Classical versus IRT
Classical Model IRT Model
� Traditional � Modern
� Requires less strict
adherence to assumptions
� Requires stricter
adherence to assumptions
� Sample dependent � Population invariant
� Statistics
(p – diff, p-biserial – disc)
� Probability-based statistics
(b-diff, a-disc, c-guessing)
� Simple scoring model (raw
score)� Scoring is more complex
2010 ICE Educational Conference
![Page 7: ABCs of IRT](https://reader034.vdocuments.net/reader034/viewer/2022052307/55895212d8b42af80b8b4586/html5/thumbnails/7.jpg)
What’s a logit?
Ability
The
Performance
StandardProbability
2010 ICE Educational Conference
![Page 8: ABCs of IRT](https://reader034.vdocuments.net/reader034/viewer/2022052307/55895212d8b42af80b8b4586/html5/thumbnails/8.jpg)
b (difficulty)
2010 ICE Educational Conference
0.00
0.10
0.20
0.30
0.40
0.50
0.60
0.70
0.80
0.90
1.00-3
-2.8
-2.5
-2.3 -2
-1.8
-1.5
-1.3 -1
-0.8
-0.5
-0.3 0
0.25 0.5
0.75 1
1.25 1.5
1.75 2
2.25 2.5
2.75
THETA
P(u
=1
| T
HE
TA
)
Paint by Numbers Leonardo
1
43
2
5
![Page 9: ABCs of IRT](https://reader034.vdocuments.net/reader034/viewer/2022052307/55895212d8b42af80b8b4586/html5/thumbnails/9.jpg)
a (discrimination) and b
2010 ICE Educational Conference
0.00
0.10
0.20
0.30
0.40
0.50
0.60
0.70
0.80
0.90
1.00-3
-2.7
5
-2.5
-2.2
5 -2
-1.7
5
-1.5
-1.2
5 -1
-0.7
5
-0.5
-0.2
5 0
0.25 0.5
0.75 1
1.25 1.5
1.75 2
2.25 2.5
2.75
THETA
P(u
=1
| T
HE
TA
)
Paint by Numbers Leonardo
1
2
3
![Page 10: ABCs of IRT](https://reader034.vdocuments.net/reader034/viewer/2022052307/55895212d8b42af80b8b4586/html5/thumbnails/10.jpg)
a, b, and c (guessing)
2010 ICE Educational Conference
0.00
0.10
0.20
0.30
0.40
0.50
0.60
0.70
0.80
0.90
1.00
-3
-2.7
5
-2.5
-2.2
5 -2
-1.7
5
-1.5
-1.2
5 -1
-0.7
5
-0.5
-0.2
5 0
0.25 0.5
0.75 1
1.25 1.5
1.75 2
2.25 2.5
2.75
THETA
P(u
=1
| T
HE
TA
)
Paint by Numbers Leonardo
1
2
3
![Page 11: ABCs of IRT](https://reader034.vdocuments.net/reader034/viewer/2022052307/55895212d8b42af80b8b4586/html5/thumbnails/11.jpg)
Fit statistics
Comparison of Infit and Outfit
0
1
2
3
4
5
6
Infit OutfitIt
em
Ord
er
ICE 2010 Conference Atlanta Georgia
Outfit Mean Square Plot
0
0.2
0.4
0.6
0.8
1
1.2
0 5 10 15 20 25 30
Item Order
MS
Q
Infit Mean Square Plot
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
0 5 10 15 20 25 30
Item Order
MS
Q
![Page 12: ABCs of IRT](https://reader034.vdocuments.net/reader034/viewer/2022052307/55895212d8b42af80b8b4586/html5/thumbnails/12.jpg)
Population Invariance
Item 3
Item 2
Item 1
.92.70
.80.60
.50.15
High
Performing
Low
Performing
Classical Difficulty Values IRT Difficulty Values
Item 3
Item 2
Item 1
-.75-.75
0.000.00
1.501.50
High
Performing
Low
Performing
2010 ICE Educational Conference
![Page 13: ABCs of IRT](https://reader034.vdocuments.net/reader034/viewer/2022052307/55895212d8b42af80b8b4586/html5/thumbnails/13.jpg)
IRT Pre-Equating
� What does it mean?
� Why would you want to do it?
� What does it mean for building item banks
and forms?
2010 ICE Educational Conference
![Page 14: ABCs of IRT](https://reader034.vdocuments.net/reader034/viewer/2022052307/55895212d8b42af80b8b4586/html5/thumbnails/14.jpg)
Test Information Function (TIF)
Comparison of Test Information Functions
0.0
10.0
20.0
30.0
40.0
50.0
60.0
70.0
-3 -2.75 -2.5 -2.25 -2 -1.75 -1.5 -1.25 -1 -0.75 -0.5 -0.25 0 0.25 0.5 0.775 1.025 1.275 1.525 1.775 2.025 2.275 2.525 2.775 3.025
Theta
Info
rmat
ion Form A
Form B
2010 ICE Educational Conference
![Page 15: ABCs of IRT](https://reader034.vdocuments.net/reader034/viewer/2022052307/55895212d8b42af80b8b4586/html5/thumbnails/15.jpg)
Assumptions
� Unidimensionality
� Local Independence
2010 ICE Educational Conference
![Page 16: ABCs of IRT](https://reader034.vdocuments.net/reader034/viewer/2022052307/55895212d8b42af80b8b4586/html5/thumbnails/16.jpg)
Implications
� Item writing� Leave those scored items alone!
� Focused item writing targeting the performance standard
� Assembly� Items selected for a form should be around the standard
� Testing and Reporting � Field test items for pre-equating/on-demand scoring
� Form assignment
� Scoring
� Recalibration
� Harder to explain to stakeholders
2010 ICE Educational Conference
![Page 17: ABCs of IRT](https://reader034.vdocuments.net/reader034/viewer/2022052307/55895212d8b42af80b8b4586/html5/thumbnails/17.jpg)
Does IRT make sense for you?
� What is the size and maturity of your program and
item bank?� Do you like to tinker with items?
� Do your program requirements change frequently?
� How experienced/capable are your item writers?
� How do you score candidates?� IRT or number correct
� Do you hold scores or do immediate scoring?
� Can you afford a psychometrician?
2010 ICE Educational Conference
![Page 18: ABCs of IRT](https://reader034.vdocuments.net/reader034/viewer/2022052307/55895212d8b42af80b8b4586/html5/thumbnails/18.jpg)
Questions?
Diane M. Talley [email protected] A. Penny [email protected] B. Johnson [email protected]
919.572.6880www.castleworldwide.com