differential item functioning in mplus
DESCRIPTION
Differential Item Functioning in Mplus. Summer School Week 2. Differential Item Functioning. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Differential Item Functioning in Mplus](https://reader033.vdocuments.net/reader033/viewer/2022061506/56813a2b550346895da21050/html5/thumbnails/1.jpg)
1
Differential Item Functioning in Mplus
Summer School Week 2
![Page 2: Differential Item Functioning in Mplus](https://reader033.vdocuments.net/reader033/viewer/2022061506/56813a2b550346895da21050/html5/thumbnails/2.jpg)
Differential Item FunctioningDifferential item functioning (DIF) occurs when people from different groups (e.g gender or ethnicity) with the same
underlying latent trait score have a different probability of responding to an item in a particular way.
Group differences in item responses (or on latent variables) do not reflect DIF per se (e.g females score higher than males on a particular item or scale).
DIF is only present if people from different groups with the same underlying ability (or trait level) have a different probability of response.
Reise, Widaman, Pugh 1993; Psych Bulletin, Vol 114, 3 552-566Embretson,S.E., Reise,S.P. (2000). Item Response Theory for Psychologists.
Definition from Laura Gibbons: ‘when a demographic characteristic interferes with relationship expected between ability level and responses to an item’
![Page 3: Differential Item Functioning in Mplus](https://reader033.vdocuments.net/reader033/viewer/2022061506/56813a2b550346895da21050/html5/thumbnails/3.jpg)
DIF – Measurement Non-Invariance
If the probablity of item response is the same (among different sub-groups with the same underlying ability) measurement invariance is assumed
If the probablity of response is different (among different sub-groups with the same underlying ability) than measurement non-invariance is assumed.
![Page 4: Differential Item Functioning in Mplus](https://reader033.vdocuments.net/reader033/viewer/2022061506/56813a2b550346895da21050/html5/thumbnails/4.jpg)
Types of DIF
Uniform: DIF occurs uniformly at all levels along the latent trait
Non-Uniform : DIF does not occur equally at all points on the latent trait (e.g. gender differences in response) may only be evident at high or low levels of the construct
Crane et al (2004) describe uniform DIF to be analogous to ‘confounding’ in epidemiology and non-uniform DIF with ‘effect modification’ – i.e. interaction between trait level, group assignment and item responses
![Page 5: Differential Item Functioning in Mplus](https://reader033.vdocuments.net/reader033/viewer/2022061506/56813a2b550346895da21050/html5/thumbnails/5.jpg)
Example of Item with Uniform DIF
From: Jones R (2006), Medical Care • Volume 44, Number 11 Suppl 3, (Figure 2)
![Page 6: Differential Item Functioning in Mplus](https://reader033.vdocuments.net/reader033/viewer/2022061506/56813a2b550346895da21050/html5/thumbnails/6.jpg)
Example of Item with Non-Uniform DIF
From: Mellenbergh, G. (1989)
![Page 7: Differential Item Functioning in Mplus](https://reader033.vdocuments.net/reader033/viewer/2022061506/56813a2b550346895da21050/html5/thumbnails/7.jpg)
7
Definition of DIF (Mellenbergh, 1989)
Item
Group
![Page 8: Differential Item Functioning in Mplus](https://reader033.vdocuments.net/reader033/viewer/2022061506/56813a2b550346895da21050/html5/thumbnails/8.jpg)
8
Definition of DIF (Mellenbergh, 1989)
P(u = 1| G, θ) = P(u = 1| θ)
An item is unbiased if...
i.e. the probability of an item response only depends on the values x of the variable X
Item
Group
Trait
![Page 9: Differential Item Functioning in Mplus](https://reader033.vdocuments.net/reader033/viewer/2022061506/56813a2b550346895da21050/html5/thumbnails/9.jpg)
9
Definition of DIF (Mellenbergh, 1989)
P(u = 1| G, θ) ≠ P(u = 1| θ)
An item is biased if...
i.e. the probability of an item response depends on the combination of values x of the variable X and values g of thevariable G
Item
Group
Trait
![Page 10: Differential Item Functioning in Mplus](https://reader033.vdocuments.net/reader033/viewer/2022061506/56813a2b550346895da21050/html5/thumbnails/10.jpg)
Differential Item Functioning
• Important first step in the evaluation of test bias• For construct validity items of a scale ideally should have little
or no DIF • Items should function in the same way across subgroups of
respondents who have the same underlying ability (or level on the latent trait)
• Presence of DIF may compromise comparison across subgroups – give misleading results
• Confound interpretation of observed variables
Camilli and Shephard, 1994
![Page 11: Differential Item Functioning in Mplus](https://reader033.vdocuments.net/reader033/viewer/2022061506/56813a2b550346895da21050/html5/thumbnails/11.jpg)
Methods to identify DIF• Parametric
– Mantel-Haenszel (MH) (Holland & Thayer, 1988) • Non-parametric methods
– Logistic regression (Zumbo, 1999) – Ordinal logistic regression (Crane et al, 2004) – MIMIC models (Muthen, 2004)– Multiple group models– IRT based methods (Thissen, 1991)
Good review by Teresi (2006) Medical Care Vol 44
![Page 12: Differential Item Functioning in Mplus](https://reader033.vdocuments.net/reader033/viewer/2022061506/56813a2b550346895da21050/html5/thumbnails/12.jpg)
What to do if DIF present• Remove items?
• 1) Ok if you have a large item pool and the item can be replaced with a item measuring similar threshold / discrimination parameters
• 2) But dropping items might adversely affect the content validity of the instrument.• 3) May end up with an instrument that is not comparable to other research using that
instrument
• Look for causes of DIF • What do all the DIF items have in common e.g.
– Are they all negatively or positively worded– Are they all at end of study – Readability etc
• How do they differ from the invariant items?
![Page 13: Differential Item Functioning in Mplus](https://reader033.vdocuments.net/reader033/viewer/2022061506/56813a2b550346895da21050/html5/thumbnails/13.jpg)
How to adjust for DIF• Adjust for DIF in the model – in Mplus can do this by
adding direct effect between the covariate and the item
• Crane et al (2004, 2006) a) items without DIF have item parameters
estimated from whole sample – (anchors) b) items with DIF have parameters estimated
separately in different subgroups
![Page 14: Differential Item Functioning in Mplus](https://reader033.vdocuments.net/reader033/viewer/2022061506/56813a2b550346895da21050/html5/thumbnails/14.jpg)
Two Examples of Identifying DIF
Mplus : MIMIC Model (Multiple Indicators, Multiple Causes) Uniform
DIF
Stata: DIFd program (Crane et al, 2004) Non-Uniform and Uniform DIF
![Page 15: Differential Item Functioning in Mplus](https://reader033.vdocuments.net/reader033/viewer/2022061506/56813a2b550346895da21050/html5/thumbnails/15.jpg)
Mplus Example - MIMIC Model:BCS70 Externalising (Conduct) Scale
03 Teenager often destroys belongings 04 Teenager frequently fights with others 10 Teenager sometimes takes others' things 14 Teenager is often disobedient 18 Teenager often tells lies 19 Teenager bullies others
Mother’s rating of teenager on Rutter Scale age 16Ordinal 3 category scale (0=does not apply, 1=applies somewhat,
2=certainly applies)
![Page 16: Differential Item Functioning in Mplus](https://reader033.vdocuments.net/reader033/viewer/2022061506/56813a2b550346895da21050/html5/thumbnails/16.jpg)
CFA Model for BCS70 Externalising
Observed items
F1 Conduct problems
RUT03
RUT04
RUT10
RUT14
RUT18
Latent Variable
.74
.91
.67
.86
.84
ε
ε
ε
ε
ε
SEX
![Page 17: Differential Item Functioning in Mplus](https://reader033.vdocuments.net/reader033/viewer/2022061506/56813a2b550346895da21050/html5/thumbnails/17.jpg)
Mimic Model Stages of identifying potential DIF
1. Run CFA model without covariates
2. Include MIMIC model (add covariate but no direct effects)
3. Add paths from covariate to indicator constrained to 0 - i.e.assuming there is no direct effect (Y1 on SEX@0)
4. Check modification indices
5. Add direct path from covariate to indicator for indicator with highest modification indices - rerun model
6. Repeat steps 4 & 5 until there are no further significant modification indices , evaluate model fit and significance of the direct effects
![Page 18: Differential Item Functioning in Mplus](https://reader033.vdocuments.net/reader033/viewer/2022061506/56813a2b550346895da21050/html5/thumbnails/18.jpg)
Stage 1-3: Mplus CFA USEVARIABLES are rut03 rut04 rut10 rut14 rut18 sex; CATEGORICAL are rut03 rut04 rut10 rut14 rut18;
Missing are all ( 88 999 );
ANALYSIS: ESTIMATOR IS wlsmv; ITERATIONS = 1000; CONVERGENCE = 0.00005;
MODEL: CONDUCT by rut03 rut04 rut10 rut14 rut18; ! (define latent variable)
OUTPUT: SAMPSTAT STANDARDIZED RES MOD(10) ;
CONDUCT on sex; ! (MIMIC model - add regression latent var on SEX rut03- rut18 on sex@0; !(assume no direct effect of sex on item)
![Page 19: Differential Item Functioning in Mplus](https://reader033.vdocuments.net/reader033/viewer/2022061506/56813a2b550346895da21050/html5/thumbnails/19.jpg)
CFA Mimic Model
Observed items
Conduct problems
SEX
RUT03
RUT04
RUT10
RUT14
RUT18
CovariateLatent Variable
On sex@0
![Page 20: Differential Item Functioning in Mplus](https://reader033.vdocuments.net/reader033/viewer/2022061506/56813a2b550346895da21050/html5/thumbnails/20.jpg)
Check MOD indices
M.I. E.P.C. Std E.P.C. StdYX E.P.C.
ON Statements RUT03 ON SEX 82.578 -0.354 -0.354 -0.176 RUT04 ON SEX 23.839 0.143 0.143 0.071
Include item with largest MI as a direct effect in model
RUT03 on SEX; RUT04- RUT18 ON SEX@0;
Recheck mod indices and repeat if necessary
![Page 21: Differential Item Functioning in Mplus](https://reader033.vdocuments.net/reader033/viewer/2022061506/56813a2b550346895da21050/html5/thumbnails/21.jpg)
Stage 4: Mplus MIMIC DIF USEVARIABLES are rut03 rut04 rut10 rut14 rut18 sex; CATEGORICAL are rut03 rut04 rut10 rut14 rut18;
Missing are all ( 88 999 );
ANALYSIS: ESTIMATOR IS wlsmv; ITERATIONS = 1000; CONVERGENCE = 0.00005;
MODEL: CONDUCT by rut03 rut04 rut10 rut14 rut18; ! (define latent variable)
RUT04 ON sex ; ! MI in run 4b 15.221 so may not be required ??
OUTPUT: SAMPSTAT STANDARDIZED RES MOD(10) ;
CONDUCT on sex; ! (MIMIC model - add regression latent var on SEX ) rut04- rut18 on sex@0; !(assume no direct effect of sex on item)
rut03 on sex; !(adds direct effect of sex on item 03)
![Page 22: Differential Item Functioning in Mplus](https://reader033.vdocuments.net/reader033/viewer/2022061506/56813a2b550346895da21050/html5/thumbnails/22.jpg)
CFA Mimic Model (DIF)
Observed EXT items
Conduct problems
SEX
RUT03
RUT04
RUT10
RUT14
RUT18
On sex@0
CovariateLatent Variable
![Page 23: Differential Item Functioning in Mplus](https://reader033.vdocuments.net/reader033/viewer/2022061506/56813a2b550346895da21050/html5/thumbnails/23.jpg)
CFA Mimic model fit1. CFA 2&3 MIMIC 4a. MIMIC
(1 direct effect) 4b. MIMIC
(2 direct effects)
Chi Square 37.9 (df=3) 131.8 (df=7) 51.9 (df=6) 38.1 (df=5)
CFI 0.997 0.989 0.996 0.997
TLI 0.994 0.986 0.994 0.995
RMSEA 0.036 0.045 0.029 0.027
WRMR 1.014 1.79 1.094 0.918
![Page 24: Differential Item Functioning in Mplus](https://reader033.vdocuments.net/reader033/viewer/2022061506/56813a2b550346895da21050/html5/thumbnails/24.jpg)
Mplus results
Model (2) Initial Mimic Model (no direct effects) Estimate S.E. Est./S.E. P-Value StdCONDUCT ON SEX -0.126 0.022 -5.789 0.000 -0.169
Model 4(b) Add 2 direct effects Estimate S.E. Est./S.E. P-Value StdCONDUCT ON SEX -0.113 0.022 -5.203 0.000 -0.152
RUT03 ON SEX -0.336 0.044 -7.597 0.000 -0.336 RUT04 ON SEX 0.112 0.032 3.481 0.000 0.112
![Page 25: Differential Item Functioning in Mplus](https://reader033.vdocuments.net/reader033/viewer/2022061506/56813a2b550346895da21050/html5/thumbnails/25.jpg)
Mplus results
Model (2) Initial Mimic Model (no direct effects) Estimate S.E. Est./S.E. P-Value StdCONDUCT ON SEX -0.126 0.022 -5.789 0.000 -0.169
Model 4(b) Add 2 direct effects Estimate S.E. Est./S.E. P-Value StdCONDUCT ON SEX -0.113 0.022 -5.203 0.000 -0.152
RUT03 ON SEX -0.336 0.044 -7.597 0.000 -0.336 RUT04 ON SEX 0.112 0.032 3.481 0.000 0.112
Is this practically meaningful?
![Page 26: Differential Item Functioning in Mplus](https://reader033.vdocuments.net/reader033/viewer/2022061506/56813a2b550346895da21050/html5/thumbnails/26.jpg)
In a Graded Response Model...
Analysis: TYPE = general missing h1 ; estimator=mlr ; algorithm=integration ;
MODEL:
RUT16EX BY rut03 rut04 rut10 rut14 rut18; ! rut19;
Rut16ex on sex;
! rut03- rut18 on sex@0;
Output: residual modindices(1.00) sampstat standardized tech1 tech5 cinterval;
![Page 27: Differential Item Functioning in Mplus](https://reader033.vdocuments.net/reader033/viewer/2022061506/56813a2b550346895da21050/html5/thumbnails/27.jpg)
In a Graded Response Model...
Analysis: TYPE = general missing h1 ; estimator=mlr ; algorithm=integration ;
MODEL:
RUT16EX BY rut03 rut04 rut10 rut14 rut18; ! rut19;
Rut16ex on sex;
! rut03- rut18 on sex@0;
Output: residual modindices(1.00) sampstat standardized tech1 tech5 cinterval;
![Page 28: Differential Item Functioning in Mplus](https://reader033.vdocuments.net/reader033/viewer/2022061506/56813a2b550346895da21050/html5/thumbnails/28.jpg)
In a Graded Response Model...
Analysis: TYPE = general missing h1 ; estimator=mlr ; algorithm=integration ;
MODEL:
RUT16EX BY rut03 rut04 rut10 rut14 rut18; ! rut19;
Rut16ex on sex;
rut03 ON sex;
rut04- rut18 on sex@0;
Output: residual modindices(1.00) sampstat standardized tech1 tech5 cinterval;
Odds ratios
![Page 29: Differential Item Functioning in Mplus](https://reader033.vdocuments.net/reader033/viewer/2022061506/56813a2b550346895da21050/html5/thumbnails/29.jpg)
ExerciseWork through MIMIC modelling stages...1. ...using multivariate probit regression model
implemented by WLSMV (equivalent to normal ogive IRT model for polyomous items)
2. ...using the Graded Response Model implemented by full-information maximum likelihood (MLR)
Note: references are included at the end of the next (DifDetect) presentation