modelling atypical students response patterns using multidimensional parametric models
DESCRIPTION
Modelling atypical students response patterns using multidimensional parametric models. Gilles Raîche, UQAM Sébastien Béland, UQAM David Magis, Université de Liège Jean-Guy Blais, Université de Montréal Pierre Brochu, CMEC Large-Scale Assessments: Policy, Research and Practice CSSE / CERA - PowerPoint PPT PresentationTRANSCRIPT
Montréal, CSSE / CERA 2010 1.Raîche, Béland, Magis, Blais and Brochu
Modelling atypical students response patterns using multidimensional parametric models
Gilles Raîche, UQAMSébastien Béland, UQAM
David Magis, Université de LiègeJean-Guy Blais, Université de Montréal
Pierre Brochu, CMEC
Large-Scale Assessments: Policy, Research and PracticeCSSE / CERA
Montréal, 2010
Montréal, CSSE / CERA 2010 2.Raîche, Béland, Magis, Blais and Brochu
• Introduction and Objectives• Unidimensional IRT Models• IRT Person Parameters Models
– Person response Curve– Multidimensional Item Response Models– Estimation– An R Package: irtProb
• Examples• Other Considerations• References and contacts
SUMMARY
Montréal, CSSE / CERA 2010 3.Raîche, Béland, Magis, Blais and Brochu
• Presentation
• IRT Models of Interest– Unidimensional latent proficiency– Dichotomous response– Monotonic– Logistic Probability Distribution
INTRODUCTION
Montréal, CSSE / CERA 2010 4.Raîche, Béland, Magis, Blais and Brochu
• Simulation of Inappropriate Response Patterns
• Person Misfit Detection Indices
• Distributional Properties of Person Misfit Indices
• Adjusted Proficiency Level Estimation in Presence of Person Misfit
OBJECTIVES
Montréal, CSSE / CERA 2010 5.Raîche, Béland, Magis, Blais and Brochu
UNIDIMENSIONAL IRT MODELS
3 Parameter Logistic (3PL) (Birnbaum, 1968)
4 Parameters Logistic (4PL) (McDonald, 1967)
where if ai is considered as a standard deviation
)(1
)(),,,|1(
iji baii
iiiijijije
cccabXPP
)(1
)1(),,,|1(
iji bai
iiiijijije
cccabXPP
2
1
i
is
a
Montréal, CSSE / CERA 2010 6.Raîche, Béland, Magis, Blais and Brochu
PERSON RESPONSE CURVE
(Trabin and Weiss, 1983)-4 -2 0 2 4
0.0
0.2
0.4
0.6
0.8
1.0
-- Difficulty (b) +
Pro
ba
bili
ty
0-200
3-100
000.30
0300.2
Montréal, CSSE / CERA 2010 7.Raîche, Béland, Magis, Blais and Brochu
MULTIDIMENSIONAL ITEM RESPONSE MODELS• Personal Variance (σ2) (Ferrando, 2004; Thurstone, 1927)
• Personal Inattention (δ)
• • Personal Pseudo-Guessing (χ) (Strandmark and Linn, 1987)
2
)(
1
),,,,,|1(
i
ij
s
b
ijiijjiiiiij
e
cdcdcbsxP
22
)(
1
),,,,,|1(
ji
ij
s
bii
ijjiiiiij
e
cdcdcbsxP
2
)(
1
),,,,,|1(
i
ij
s
b
jiijijjiiiiij
e
cdcdcbsxP
Montréal, CSSE / CERA 2010 8.Raîche, Béland, Magis, Blais and Brochu
MULTIDIMENSIONAL ITEM RESPONSE MODELS• Higher Order Models
22
)(
1
),,,,,,|1() and ,(
ji
ij
s
b
jiiijjjiiiiij
e
cdcdcbsxP
22
)(
1
)(),,,,,,|1() and ,(
ji
ij
s
b
ijiijjjiiiiij
e
cdcdcbsxP
22
)(
1
)(),,,,,,,|1() and ,,(
ji
ij
s
b
jijiijjjjiiiiij
e
cdcdcbsxP
Montréal, CSSE / CERA 2010 9.Raîche, Béland, Magis, Blais and Brochu
• Package: irtProb• MAP Estimators• A Priori Probability Distribution
– σ : U(0,4) – θ: U(-4,4)– X: U(0,1)– δ : U(0,1)
ESTIMATION OF SUBJECT PARAMETERS
Montréal, CSSE / CERA 2010 10.Raîche, Béland, Magis, Blais and Brochu
• Available on R Cran Site• Functionnalities
– Estimation of Person Parameters (MAP)– Likelihood Curves– Person Characteristic Curves– Probability, Density and Random Functions– Simulation of Response Patterns– Classical <-> IRT Item Parameters– Model Selection
A R PACKAGE: irtProb
Montréal, CSSE / CERA 2010 11.Raîche, Béland, Magis, Blais and Brochu
EXAMPLES – 01 (X)
Table 1. Person Parameter estimation from 100 simulated subjects attempting to augment their estimated proficiency level to a 40 items test [person parameter (Standard Error)]
Pseudo-Guessing
Model1 0.00 0.10 0.20 0.30 0.40
1 θ = -2 -2.18 (0.62)
-1.28 (0.69)
-0.41 (0.69)
0.03 (0.83)
1.00 (0.72)
2 θ = -2C
-2.18 -2.18 (0.62)(0.62)
0.00 0.00 (0.01)(0.01)
-1.75 -1.75 (0.69)(0.69)
0.06 0.06 (0.07)(0.07)
-1.55 -1.55 (0.70)(0.70)
0.16 0.16 (0.10)(0.10)
-1.96 -1.96 (0.88)(0.88)
0.26 0.26 (0.12)(0.12)
-1.64 -1.64 (1.18)(1.18)
0.35 0.35 (0.13)(0.13)
3 θ = -2CSD
-1.98 (0.64)0.01
(0.02)0.07
(0.23)0.03
(0.08)
-1.61 -1.61 (0.74)(0.74)
0.07 0.07 (0.08)(0.08)
0.17 0.17 (0.44)(0.44)
0.03 0.03 (0.07)(0.07)
-1.46 -1.46 (0.79)(0.79)
0.17 0.17 (0.11)(0.11)
0.23 0.23 (0.51)(0.51)
0.03 0.03 (0.06)(0.06)
-1.62 -1.62 (1.10)(1.10)
0.24 0.24 (0.14)(0.14)
0.44 0.44 (0.92)(0.92)
0.03 0.03 (0.07)(0.07)
-1.22 -1.22 (1.59)(1.59)
0.33 0.33 (0.17)(0.17)
0.43 0.43 (0.87)(0.87)
0.02 0.02 (0.05)(0.05)
1 σ = 0, δ = 0, b = -5 to 5, c = 0, d = 0, 40 items, 100 simulated sujects Model 1: θ only Model 2: θ and Pseudo-Guessing Model 3 σ, θ, Pseudo-Guessing and δ
Montréal, CSSE / CERA 2010 12.Raîche, Béland, Magis, Blais and Brochu
EXAMPLES – 01 (X)
(θ=-2, X=0.2) (θ=-2, X=0.4)
-4
-2
0
2
4
0.0
0.2
0.4
0.6
0.8
1.0
0e+00
2e-11
4e-11
6e-11
8e-11
P(X)
-4
-2
0
2
4
0.0
0.2
0.4
0.6
0.8
1.0
0.0e+00
5.0e-11
1.0e-10
1.5e-10
2.0e-10
2.5e-10
P(X)
Montréal, CSSE / CERA 2010 13.Raîche, Béland, Magis, Blais and Brochu
EXAMPLES – 02 (X)
Table 2. Person Parameter estimation from 100 simulated subjects attempting to augment their estimated proficiency level to a 40 items test [person parameter (Standard Error)]
Pseudo-Guessing
Model1 0.00 0.10 0.20 0.30 0.40
1 θ = 2 2.19 (0.55) 2.52 (0.61) 3.02 (0.58) 3.31 (0.62) 3.46 (0.56)
2 θ = 2
C
2.09 (0.61)
0.02 (0.06)
2.37 (0.60)2.37 (0.60)
0.04 (0.10)
2.52 (0.85)
0.12 (0.18)
2.74 (0.88)
0.16 (0.21)
2.51 (1.01)
0.27 (0.23)
3 θ = 2
S
C
D
1.95 (0.55)
0.11 (0.27)
0.02 (0.05)
0.01 (0.02)
2.11 (0.61)2.11 (0.61)
0.17 (0.34)0.17 (0.34)
0.08 (0.13)0.08 (0.13)
0.00 (0.01)0.00 (0.01)
2.10 (0.93)2.10 (0.93)
0.12 (0.32)0.12 (0.32)
0.20 (0.19)0.20 (0.19)
0.00 (0.01)0.00 (0.01)
2.42 (0.90)2.42 (0.90)
0.12 (0.40)0.12 (0.40)
0.22 (0.21)0.22 (0.21)
0.00 (0.01)0.00 (0.01)
2.02 (1.08)2.02 (1.08)
0.09 (0.30)0.09 (0.30)
0.35 (0.23)0.35 (0.23)
0.00 (0.01)0.00 (0.01)1 σ = 0, δ = 0, b = -5 to 5, c = 0, d = 0, 40 items, 100 simulated subjects Model 1: θ only Model 2: θ and Pseudo-Guessing Model 3 σ, θ, Pseudo-Guessing and δ
Montréal, CSSE / CERA 2010 14.Raîche, Béland, Magis, Blais and Brochu
EXAMPLES – 02 (X)
(θ=2, X=0.2) (θ=2, X=0.4)
-4
-2
0
2
4
0.0
0.2
0.4
0.6
0.8
1.0
0e+00
2e-05
4e-05
6e-05
8e-05
P(X)
-4
-2
0
2
4
0.0
0.2
0.4
0.6
0.8
1.0
0.0e+00
5.0e-06
1.0e-05
1.5e-05
2.0e-05
2.5e-05
3.0e-05
P(X)
Montréal, CSSE / CERA 2010 15.Raîche, Béland, Magis, Blais and Brochu
EXAMPLES – 03 (σ)
Table 3. Person Parameter estimation from 100 simulated subjects with fluctuating proficiency level to a 40 items test [person parameter (Standard Error)]
Fluctuation
Model1 0.00 0.50 1.00 2.00 4.00
1 θ = -2 -2.14 (0.59) -2.07 (0.61) -2.00 (0.62) -1.71 (0.70) -1.01 (0.86)
2 θ = -2
S
-2.06 (0.58)
0.33 (0.42)
-2.03 (0.59)
0.51 (0.57)
-2.05 (0.67)
0.90 (0.68)
-2.00 (0.84)
1.87 (0.90)
-1.63 (1.28)
3.45 (0.75)
3 θ = -2
S
C
D
-1.92 (0.66)
0.11 (0.24)
0.01 (0.01)
0.04 (0.09)
-1.75 (0.79)
0.15 (0.29)
0.01 (0.02)
0.06 (0.11)
-1.54 (0.83)
0.28 (0.45)
0.01 (0.03)
0.11 (0.14)
-0.97 (1.42)
0.71 (0.84)
0.03 (0.05)
0.18 (0.18)
-0.68 (2.37)
0.79 (1.24)
0.15 (0.14)
0.23 (0.21)1 X = 0, δ = 0, b = -5 to 5, c = 0, d = 0, 40 items, 100 simulated subjects Model 1: θ only Model 2: θ and σ Model 3 σ, θ, Pseudo-Guessing and δ
Montréal, CSSE / CERA 2010 16.Raîche, Béland, Magis, Blais and Brochu
EXAMPLES – 03 (σ)
(θ=-2, σ=1) (θ=2, σ=4)
-4
-2
0
2
4
0
1
2
3
4
2.0e-12
4.0e-12
6.0e-12
8.0e-12
1.0e-11
1.2e-11
P(X)
-4
-2
0
2
4
0
1
2
3
4
2e-06
4e-06
6e-06
8e-06
P(X)
Montréal, CSSE / CERA 2010 17.Raîche, Béland, Magis, Blais and Brochu
OTHER CONSIDERATIONS
• Multidimensional EAP Estimation Very Computer Intensive
• Warm Weighted Likelihood Estimator
• Item Parameters Estimation
• Confidence Interval For The Additionnal Person Parameters
• Other Person Fit Indices: Pseudo-Guessing and Inattention
Montréal, CSSE / CERA 2010 18.Raîche, Béland, Magis, Blais and Brochu
Barton, M. A. and Lord, F. M. (1981). An upper asymptote for the three-parameter logistic item-response model. Research bullelin 81-20. Princeton, NJ: Educational Testing Service.
Birnbaum, A. (1968). Some latent trait models and their use in inferring an examinee’s ability. In F. M. Lord and M. Novick (Eds): Statistical theories of mental test scores. New York, NJ: Addison-Wesley.
Ferrando, P. J. (2004). Person reliability in personality measurement: an item response theory analysis. Applied Psychological Measurement, 28(2), 126-140.
Hulin, C. L., Drasgow, F., and Parsons, C. K. (1983). Item response theory. Homewood, IL: Irwin.
Levine, M. V., and Drasgow, F. (1983). Appropriateness measurement: validating studies and variable ability models. In D. J. Weiss (Ed.): New horizons in testing. New York, NJ: Academic Press.
Magis, D. (2007). Enhanced estimation methods in IRT. In D. Magis (Ed.): Influence, information and item response theory in discrete data analysis. Doctoral dissertation, Liège, Belgium: University de Liège.
REFERENCES / 1
Montréal, CSSE / CERA 2010 19.Raîche, Béland, Magis, Blais and Brochu
McDonald, R. P. (1967). Nonlinear factor analysis. Psyhometric Monographs, 15.
Raîche, G., and Blais, J.-G. (2003). Efficacité du dépistage des étudiants et des étudiants qui cherchent à obtenir un résultat faible au test de classement en anglais, langue seconde, au collégial. In J.-G. Blais, and G. Raîche (Ed.): Regards sur la modélisation de la mesure en en éducation et en sciences sociales. Ste-Foy, QC: Presses de l’Université Laval.
Strandmark, N. L. and Linn, R. L. (1987). A generalized logistic item response model parameterizing test score inappropriateness. Applied Psychological Measurement, 11(4), 355-370.
Thurstone, L. L. (1927). A law of comparative judgment. Psychological Review, 34, 273-286.
Trabin, T. E., and Weiss, D. J. (1983). The person response curve : fit of individuals to item response theory models. In D. J. Weiss (Ed.): New horizons in testing. New York, NJ: Academic Press.
REFERENCES / 2
Montréal, CSSE / CERA 2010 20.Raîche, Béland, Magis, Blais and Brochu
• Gilles Raîche– http://camri.uqam.ca
• Sébastien Béland– [email protected]
• David Magis– [email protected]
• Jean-Guy Blais– http://www.griemetic.ca
• Pierre Brochu– [email protected]
CONTACTS