detecting dif/dsf with pcmtrees detecting di erential item...

Detecting

DIF/DSF with

PCMtrees

Testing for DIF in

the RM

Standard tests

Model-based recursive

partitioning

Extension to the

PCM

DIF/DSF in the PCM

(Un)ordered threshold

parameters

Visualization

Example: Verbal

Aggression data

Summary

References

Detecting Differential Item and

Differential Step Functioning with

Partial Credit Trees

Basil Abou El-Komboz, Achim Zeileis and Carolin Strobl

Detecting

DIF/DSF with

PCMtrees

Testing for DIF in

the RM

Standard tests


partitioning

Extension to the

PCM

DIF/DSF in the PCM


parameters

Visualization

Example: Verbal

Aggression data

Summary

References

Outline

Testing for DIF in the Rasch model

Standard model tests

Model-based recursive partitioning

Extending the model-based recursive partitioning approach

to the Partial Credit Model (PCM)

Differential item and step functioning in the PCM

(Un)ordered threshold parameters in the PCM

Visualization in Partial Credit trees

Example: Verbal Aggression data

Summary

Detecting

DIF/DSF with

PCMtrees

Testing for DIF in

the RM

Standard tests


partitioning

Extension to the

PCM

DIF/DSF in the PCM


parameters

Visualization

Example: Verbal

Aggression data

Summary

References

Differential Item Functioning (DIF)

is present when one or more items of a test

I are easier or harder to solve for certain subjects

I even though they have the same latent trait

Detecting

DIF/DSF with

PCMtrees

Testing for DIF in

the RM

Standard tests


partitioning

Extension to the

PCM

DIF/DSF in the PCM


parameters

Visualization

Example: Verbal

Aggression data

Summary

References


I tests for k given groups

graphical test, Andersen’s Likelihood-Ratio Test,

Wald Tests

+ straightforward interpretation

− only detect DIF in specified groups

I latent-class approach

Rost’s “Mixed” (mixture) Rasch model

+ identifies previously unknown groups with DIF

− groups are not directly interpretable

⇒ 2nd step: describe groups with covariates

(e.g., Cohen and Bolt, 2005)

Detecting

DIF/DSF with

PCMtrees

Testing for DIF in

the RM

Standard tests


partitioning

Extension to the

PCM

DIF/DSF in the PCM


parameters

Visualization

Example: Verbal

Aggression data

Summary

References


●

●●●

●

●

●

●●

●

●

●

−3 −2 −1 0 1 2 3

−3

−2

−1

01

23

Geschlecht = Mann

Ges

chle

cht =

Fra

u 1

234

5

6

7

89

10

11

12

●

●●

●

●

●

●

●

●

●

●

●

(Mair, Hatzinger, and Maier, 2010, package eRm)

Detecting

DIF/DSF with

PCMtrees

Testing for DIF in

the RM

Standard tests


partitioning

Extension to the

PCM

DIF/DSF in the PCM


parameters

Visualization

Example: Verbal

Aggression data

Summary

References


I tests for k given groups

graphical test, Andersen’s Likelihood-Ratio Test,

Wald Tests

+ straightforward interpretation

− only detect DIF in specified groups

I latent-class approach

Rost’s “Mixed” (mixture) Rasch model


− groups are not directly interpretable

⇒ 2nd step: describe groups with covariates

(e.g., Cohen and Bolt, 2005)

Detecting

DIF/DSF with

PCMtrees

Testing for DIF in

the RM

Standard tests


partitioning

Extension to the

PCM

DIF/DSF in the PCM


parameters

Visualization

Example: Verbal

Aggression data

Summary

References

New: Model-based recursive partitioning


+ straightforward interpretationgender

p = 0.006

1

male female

agep < 0.001

2

≤ 34 > 34

Node 3 (n = 35)

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

● ●

●●

●

1 20

−2.68

4.66Node 4 (n = 74)

●

● ● ●

●

● ●

●

●

●

●● ●

●●

●

●

●

●

●

1 20

−2.68

4.66Node 5 (n = 91)

● ●

●

●

●

●

●

●

●

● ●

●

●

●●

●

●

●

●●

1 20

−2.68

4.66

function raschtree in package psychotree

Detecting

DIF/DSF with

PCMtrees

Testing for DIF in

the RM

Standard tests


partitioning

Extension to the

PCM

DIF/DSF in the PCM


parameters

Visualization

Example: Verbal

Aggression data

Summary

References

Approach used in psychotree takes care of...

I selecting splitting variables ⇔ parameter instability tests

scor

e co

ntrib

utio

ns

25 30 35 40 45

−0.

40

0.2

0.4

age

I selecting optimal cutpoints

I other multiple testing issues

I between variables in each split

I over successive splits

(Zeileis and Hornik, 2007; Zeileis, Hothorn, and Hornik, 2008;

Strobl, Malley, and Tutz, 2009; Strobl, Kopf, and Zeileis, 2010a,b)

Detecting

DIF/DSF with

PCMtrees

Testing for DIF in

the RM

Standard tests


partitioning

Extension to the

PCM

DIF/DSF in the PCM


parameters

Visualization

Example: Verbal

Aggression data

Summary

References

Extending the model-based partitioning approach

Rasch trees genderp = 0.006

1

male female

agep < 0.001

2

≤ 34 > 34

Node 3 (n = 35)

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

● ●

●●

●

1 20

−2.68

4.66Node 4 (n = 74)

●

● ● ●

●

● ●

●

●

●

●● ●

●●

●

●

●

●

●

1 20

−2.68

4.66Node 5 (n = 91)

● ●

●

●

●

●

●

●

●

● ●

●

●

●●

●

●

●

●●

1 20

−2.68

4.66

Partial Credit trees

Detecting

DIF/DSF with

PCMtrees

Testing for DIF in

the RM

Standard tests


partitioning

Extension to the

PCM

DIF/DSF in the PCM


parameters

Visualization

Example: Verbal

Aggression data

Summary

References


Rasch trees genderp = 0.006

1

male female

agep < 0.001

2

≤ 34 > 34

Node 3 (n = 35)

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

● ●

●●

●

1 20

−2.68

4.66Node 4 (n = 74)

●

● ● ●

●

● ●

●

●

●

●● ●

●●

●

●

●

●

●

1 20

−2.68

4.66Node 5 (n = 91)

● ●

●

●

●

●

●

●

●

● ●

●

●

●●

●

●

●

●●

1 20

−2.68

4.66

Partial Credit trees

Detecting

DIF/DSF with

PCMtrees

Testing for DIF in

the RM

Standard tests


partitioning

Extension to the

PCM

DIF/DSF in the PCM


parameters

Visualization

Example: Verbal

Aggression data

Summary

References


Rasch model

I scores are 0 or 1

I each item has one location parameter = difficulty

I DIF means item is more/less difficult for certain group

Partial Credit model

I scores are between 0 and mj

I different parametrizations: e.g. mj thresholds

I DIF means entire item is more/less difficult

I DSF means some steps are more/less difficult

(may cancel out so there is no overall DIF)

Detecting

DIF/DSF with

PCMtrees

Testing for DIF in

the RM

Standard tests


partitioning

Extension to the

PCM

DIF/DSF in the PCM


parameters

Visualization

Example: Verbal

Aggression data

Summary

References


−5 0 5 10 15

0.0

0.2

0.4

0.6

0.8

1.0

P

(uij=

c|θ i

,δj1,..

.,δj3)

δj1 δj2 δj3

0 1 2 3

P(uij = c |θi , δj1, . . . , δjmj) =

e∑c

k=0(θi−δjk)

∑mj

l=0 e∑l

k=0(θi−δjk)

with∑0

k=0 (θi − δjk) = 0

Detecting

DIF/DSF with

PCMtrees

Testing for DIF in

the RM

Standard tests


partitioning

Extension to the

PCM

DIF/DSF in the PCM


parameters

Visualization

Example: Verbal

Aggression data

Summary

References


−5 0 5 10 15

0.0

0.2

0.4

0.6

0.8

1.0

P(u

ij=c|

θ i,δ

j1,..

.,δj3)

δj1δj2 δj3

0 1

2 3

P(uij = c |θi , δj1, . . . , δjmj) =

e∑c

k=0(θi−δjk)

∑mj

l=0 e∑l

k=0(θi−δjk)

with∑0

k=0 (θi − δjk) = 0

Detecting

DIF/DSF with

PCMtrees

Testing for DIF in

the RM

Standard tests


partitioning

Extension to the

PCM

DIF/DSF in the PCM


parameters

Visualization

Example: Verbal

Aggression data

Summary

References


Want Curse

0

0.2

0.4

0.6

0.8

1

δ1δ1 δ2δ2

Do Curse

0

0.2

0.4

0.6

0.8

1

δ1δ1δ2δ2

Want Scold

0

0.2

0.4

0.6

0.8

1

δ1δ1δ2δ2

Do Scold

0

0.2

0.4

0.6

0.8

1

δ1δ1δ2δ2

Want Shout

0

0.2

0.4

0.6

0.8

1

δ1δ1 δ2δ2

Do Shout

0

0.2

0.4

0.6

0.8

1

δ1δ1δ2δ2

−2 −1 0 1 2

−2 −1 0 1 2

Category Characteristic Curves

Pro

babi

lity

Latent Trait

Cat. 0 Cat. 1 Cat. 2

Detecting

DIF/DSF with

PCMtrees

Testing for DIF in

the RM

Standard tests


partitioning

Extension to the

PCM

DIF/DSF in the PCM


parameters

Visualization

Example: Verbal

Aggression data

Summary

References


Want Shout

0

0.2

0.4

0.6

0.8

1

δ1δ1 δ2δ2

Do Shout

0

0.2

0.4

0.6

0.8

1

δ1δ1δ2δ2

−2 −1 0 1 2 3

−2 −1 0 1 2 3


Prob

abilit

y

Latent Trait


Detecting

DIF/DSF with

PCMtrees

Testing for DIF in

the RM

Standard tests


partitioning

Extension to the

PCM

DIF/DSF in the PCM


parameters

Visualization

Example: Verbal

Aggression data

Summary

References


Want Curse

0

0.2

0.4

0.6

0.8

1

δ1δ1 δ2δ2

Do Curse

0

0.2

0.4

0.6

0.8

1

δ1δ1δ2δ2

Want Scold

0

0.2

0.4

0.6

0.8

1

δ1δ1δ2δ2

Do Scold

0

0.2

0.4

0.6

0.8

1

δ1δ1δ2δ2

Want Shout

0

0.2

0.4

0.6

0.8

1

δ1δ1 δ2δ2

Do Shout

0

0.2

0.4

0.6

0.8

1

δ1δ1δ2δ2

−2 −1 0 1 2

−2 −1 0 1 2


Pro

babi

lity

Latent Trait


Detecting

DIF/DSF with

PCMtrees

Testing for DIF in

the RM

Standard tests


partitioning

Extension to the

PCM

DIF/DSF in the PCM


parameters

Visualization

Example: Verbal

Aggression data

Summary

References


Wan

t Cur

se

0

0.2

0.4

0.6

0.81

δ 1δ 1δ 2δ 2

Do

Cur

se

00.2

0.4

0.6

0.8

1

δ 1δ 1δ 2δ 2

Wan

t Sco

ld

0

0.2

0.4

0.6

0.81

δ 1δ 1δ2δ 2

Do

Sco

ld

00.2

0.4

0.6

0.8

1

δ 1δ 1δ 2δ 2

Wan

t Sho

ut

0

0.2

0.4

0.6

0.81

δ 1δ 1δ 2δ 2

Do

Sho

ut

00.2

0.4

0.6

0.8

1

δ 1δ 1δ 2δ 2

−2

−1

01

2

−2

−1

01

2

Cat

egor

y C

hara

cter

istic

Cur

ves

Probability

Late

nt T

rait

Cat

. 0C

at. 1

Cat

. 2

Late

nt tr

ait

−2

−1

01

2

−2

−1

01

2

Want Curse Do Curse Want Scold Do Scold Want Shout Do Shout

inspired by “effect plots”

(Fox and Hong, 2009, package effects)

Detecting

DIF/DSF with

PCMtrees

Testing for DIF in

the RM

Standard tests


partitioning

Extension to the

PCM

DIF/DSF in the PCM


parameters

Visualization

Example: Verbal

Aggression data

Summary

References


Wan

t Cur

se

0

0.2

0.4

0.6

0.81

δ 1δ 1δ 2δ 2

Do

Cur

se

00.2

0.4

0.6

0.8

1

δ 1δ 1δ 2δ 2

Wan

t Sco

ld

0

0.2

0.4

0.6

0.81

δ 1δ 1δ2δ 2

Do

Sco

ld

00.2

0.4

0.6

0.8

1

δ 1δ 1δ 2δ 2

Wan

t Sho

ut

0

0.2

0.4

0.6

0.81

δ 1δ 1δ 2δ 2

Do

Sho

ut

00.2

0.4

0.6

0.8

1

δ 1δ 1δ 2δ 2

−2

−1

01

2

−2

−1

01

2

Cat

egor

y C

hara

cter

istic

Cur

ves

Probability

Late

nt T

rait

Cat

. 0C

at. 1

Cat

. 2

Late

nt tr

ait

−2

−1

01

2

−2

−1

01

2

Want Curse Do Curse Want Scold Do Scold Want Shout Do Shout

inspired by “effect plots”

(Fox and Hong, 2009, package effects)

Detecting

DIF/DSF with

PCMtrees

Testing for DIF in

the RM

Standard tests


partitioning

Extension to the

PCM

DIF/DSF in the PCM


parameters

Visualization

Example: Verbal

Aggression data

Summary

References


> data("VerbalAggression", package = "psychotools")

responses of 316 subjects to frustrating situations

I here: situation 4 (self-to-blame situation)

“The operator disconnects me when I used up my last 10

cents for a call.”

I items: 3 verbally aggressive responses (curse, scold, shout)

× 2 behavioural models (want, do)

I response categories: 0 = no, 1 = perhaps, 2 = yes

I covariates: gender, trait anger (assessed by the Dutch

adaptation of the state-trait anger scale STAS)

De Boeck and Wilson (2004), Smits, De Boeck, and Vansteelandt

(2004), dichotomized version also available in package difR

(Magis, Beland, and Raiche, 2011)

Detecting

DIF/DSF with

PCMtrees

Testing for DIF in

the RM

Standard tests


partitioning

Extension to the

PCM

DIF/DSF in the PCM


parameters

Visualization

Example: Verbal

Aggression data

Summary

References


genderp = 0.027

1

female male

Node 2 (n = 243)

WantCurse

DoCurse

WantScold

DoScold

WantShout

DoShout

−2

−1

0

1

2

Late

nt tr

ait

angerp = 0.198

3

≤ 21 > 21

Node 4 (n = 49)

WantCurse

DoCurse

WantScold

DoScold

WantShout

DoShout

−2

−1

0

1

2

Late

nt tr

ait

Node 5 (n = 24)

WantCurse

DoCurse

WantScold

DoScold

WantShout

DoShout

−2

−1

0

1

2

Late

nt tr

ait

Partial Credit tree (tweaked a little for visualization)

Detecting

DIF/DSF with

PCMtrees

Testing for DIF in

the RM

Standard tests


partitioning

Extension to the

PCM

DIF/DSF in the PCM


parameters

Visualization

Example: Verbal

Aggression data

Summary

References

Summary

model-based recursive partitioning

I can identify groups of subjects with DIF and DSF that

I need not be pre-specified

I are formed by (combinations of) observed covariates

I with optimally selected cutpoints

I available for

I Rasch model

I Partial Credit model

I and more to come

I results are directly interpretable

, but keep in mind:

observed covariates may be proxies for the true causes

e.g.: gender ⇔ socialization, district ⇔ first language

Detecting

DIF/DSF with

PCMtrees

Testing for DIF in

the RM

Standard tests


partitioning

Extension to the

PCM

DIF/DSF in the PCM


parameters

Visualization

Example: Verbal

Aggression data

Summary

References

Summary






I available for

I Rasch model


I and more to come


, but keep in mind:



Detecting

DIF/DSF with

PCMtrees

Testing for DIF in

the RM

Standard tests


partitioning

Extension to the

PCM

DIF/DSF in the PCM


parameters

Visualization

Example: Verbal

Aggression data

Summary

References

Summary






I available for

I Rasch model


I and more to come


, but keep in mind:



Detecting

DIF/DSF with

PCMtrees

Testing for DIF in

the RM

Standard tests


partitioning

Extension to the

PCM

DIF/DSF in the PCM


parameters

Visualization

Example: Verbal

Aggression data

Summary

References

Summary






I available for

I Rasch model


I and more to come

I results are directly interpretable, but keep in mind:



Detecting

DIF/DSF with

PCMtrees

Testing for DIF in

the RM

Standard tests


partitioning

Extension to the

PCM

DIF/DSF in the PCM


parameters

Visualization

Example: Verbal

Aggression data

Summary

References

References I

Cohen, A. and D. Bolt. “A Mixture Model Analysis of

Differential Item Functioning.” Journal of Educational

Measurement 42 (2005): 133–148.

De Boeck, P. and M. Wilson, editors. Explanatory Item

Response Models: A Generalized Linear and Nonlinear

Approach. New York: Springer, 2004.

Fox, John and Jangman Hong. “Effect Displays in R for

Multinomial and Proportional-Odds Logit Models:

Extensions to the effects Package.” Journal of

Statistical Software 32 (2009): 1–24.

Detecting

DIF/DSF with

PCMtrees

Testing for DIF in

the RM

Standard tests


partitioning

Extension to the

PCM

DIF/DSF in the PCM


parameters

Visualization

Example: Verbal

Aggression data

Summary

References

References II

Magis, D., S. Beland, and G. Raiche. difR: Collection of

methods to detect dichotomous differential item

functioning (DIF) in psychometrics, 2011. R package

version 4.1.

Mair, Patrick, Reinhold Hatzinger, and Marco Maier. eRm:

Extended Rasch Modeling., 2010. R package version

0.13-0.

Smits, D., P. De Boeck, and K. Vansteelandt. “Inhibition of

Verbally Aggressive Behaviour.” European Journal of

Personality 18 (2004).

Detecting

DIF/DSF with

PCMtrees

Testing for DIF in

the RM

Standard tests


partitioning

Extension to the

PCM

DIF/DSF in the PCM


parameters

Visualization

Example: Verbal

Aggression data

Summary

References

References III

Strobl, C., J. Kopf, and A. Zeileis. A New Method for

Detecting Differential Item Functioning in the Rasch

Model. Technical Report 92, Department of Statistics,

Ludwig-Maximilians-Universitat Munchen, Germany, 2010.

URL: http://epub.ub.uni-muenchen.de/11915/.

Strobl, C., J. Kopf, and A. Zeileis. “Wissen Frauen weniger

oder nur das Falsche? – Ein statistisches Modell fur

unterschiedliche Aufgaben-Schwierigkeiten in

Teilstichproben.” Allgemeinbildung in Deutschland –

Erkenntnisse aus dem SPIEGEL Studentenpisa-Test. Ed.

S. Trepte and M. Verbeet Wiesbaden: VS Verlag, 2010,

255–272.

Detecting

DIF/DSF with

PCMtrees

Testing for DIF in

the RM

Standard tests


partitioning

Extension to the

PCM

DIF/DSF in the PCM


parameters

Visualization

Example: Verbal

Aggression data

Summary

References

References IV

Strobl, C., J. Malley, and G. Tutz. “An Introduction to

Recursive Partitioning: Rationale, Application and

Characteristics of Classification and Regression Trees,

Bagging and Random Forests.” Psychological Methods 14

(2009): 323–348.

Zeileis, A., T. Hothorn, and K. Hornik. “Model-Based

Recursive Partitioning.” Journal of Computational and

Graphical Statistics 17 (2008): 492–514.

Zeileis, Achim and Kurt Hornik. “Generalized M-Fluctuation

Tests for Parameter Instability.” Statistica Neerlandica 61

(2007): 488–508.

detecting dif/dsf with pcmtrees detecting di erential item...

Documents