evaluating sensitivity of parameters of interest to ... · references conclusion • this talk...

References

Evaluating sensitivity of parametersof interest to measurement invariance

using the EPC-interest

Daniel Oberski

Department of methodology and statistics, Tilburg University

WorkingGroupStructuralEquationModeling26-27.02.2015, FU Berlin

Measurement invariance using the EPC-interest Daniel Oberski

References

Conclusion

• This talk discusses the ``EPC-interest''• EPC-interest is like SEM's expected parameter change

(``EPC-self'') but instead of measuring the change in therestricted parameter, it measures the change in theparameter of interest;

• Introduced for measurement invariance evaluation incontinuous data SEM by Oberski (2014) and forcategorical data by Oberski et al. (frth)

• Implemented in Latent Gold ≥5.0, experimental version oflavaan, working on main branch.


References

The problem of measurement invariance


References

p(y) =∑ξ

p(ξ|x)J∏j=1

p(yj|ξ)


References

p(y) =∑ξ

p(ξ|x)J∏j=1

p(yj|ξ, x)


References

Measurement invariance: the problem

Problem of measurement invariance: we want to know γ,but δ = 0 might bias this parameterofinterest.


References

Preceding solutions to the problem

..1 Selecting one indicator as a reference indicator (``anchoritem'') apriori;

..2 Imposing a strong prior on differential functioning(Muthén&Asparouhov2012)

..3 Test null hypothesis of full or partial invariance.

..4 Sensitivity analysis: allow partial violations when theymatter.


References



+ Don't need further restrictions;- No way of testing reference indicator.

E.g.: Setting loadings to 1 in each group, ``alignment method''.





References




+ Attractive data-driven solution when prior is neither tooinformative nor too weak;

- More research needed to figure out when prior is neithertoo informative nor too weak.




References

Preceding solutions to the problem..1 Selecting one indicator as a reference indicator (``anchor

item'') apriori;..2 Imposing a strong prior on differential functioning(Muthén&Asparouhov2012)

..3 Test null hypothesis of full or partial invariance.+ When high-powered test of full measurement invariance is

not rejected, may be safe to simply continue without theneed for further modeling;

- Rarely happens in practice.+ Partial invariance looks at size and significance of δ, but

not all big δ's are important nor are small ones necessarilyunimportant. So does not guarantee that he parameter ofinterest in free of measurement differences (Oberski,2014)



References







References

Measurement invariance: the problem and a solution

Problem of measurement invariance: we want to know γ,but δ = 0 might bias this parameterofinterest.

Solution: Use EPC-interest: the expected change in γ whenfreeing δ.

• If EPC-interest is big (e.g. can change sign), incorporate δ;• if EPC-interest is small, ignore it.


References

EPC-interest


References

EPC-interest, also for categorical data

EPC-interest = P(∂θ

∂δ′

)(δ − δ

)= −PH−1

θθ Hθδ EPC-self= γa − γ +O(ψ′ψ),

where P selects the parameters of interest γ from theparameter vector θ, H is a Hessian, and O(ψ′ψ) is anapproximation term depending on the overall amount ofmisspecification (parameter differences).


References

Simulation: how good is the approximation?Setup:

P(Yj = 1|x) = [1 + exp(−x)]−1,

with j ∈ {2, 3, 4}, and structural model

x = γz+ ϵ

with γ = 1 and ϵ ∼ N(0, 1). We then introduced a violation ofmeasurement invariance for the first indicator,

P(Y1 = 1|x) = [1 + exp(−x− δz)]−1.

Nine conditions varied sample size, n ∈ {250, 500, 1000}, andthe size of the invariance violation: δ = 0 (no violation), 0.5(moderate), or 1 (extreme). Data were generated using R 3.1.2and analyzed using Latent GOLD 5.0.0.14161.


References

n 250 500 1000True δ 0 0.5 1 0 0.5 1 0 0.5 1Est. γ 1.010 1.151 1.353 0.980 1.152 1.330 1.013 1.163 1.327Bias γ -0.010 -0.151 -0.353 0.020 -0.152 -0.330 -0.013 -0.163 -0.327EPC-int. 0.003 -0.166 -0.494 -0.001 -0.180 -0.486 0.004 -0.182 -0.448

Table : Simulation study of EPC-interest. Shown is the average pointestimate for the γ parameter of interest under full measurementinvariance (``Est''), its difference from the true value γ = 1 (``Bias''),and the average EPC-interest.


References

Example with categorical dataSee http://daob.nl/publications/


http://daob.nl/publications/

References

Figure : Graphical representation of the multilevel latent classregression model for (post)materialism measured by three partialranking tasks. Observed variables are shown in rectangles whileunobserved (``latent'') variables are shown in ellipses.


References

Multilevel latent class model w/ covariates forrankings

L(θ) = P(A1,A2,B1,B2,C1,C2|Z1,Z2) =

C∏c=1

∑G

P(Gc)nc∏i=1

∑X

P(Xic|Z1ic,Z2ic,Gc)×

P(A1ic,A2ic|Xic)P(B1ic,B2ic|Xic)P(C1ic,C2ic|Xic),

Goal: estimate γ (especially its sign).Possibleproblem: Violations of scalar and metricmeasurement invariance (DIF), parameterized respectivelyas τ∗ and λ∗.Solution: See if these matter for the sign of γ.


References

Table : Full invariance multilevel latent class model: EPC-interestvalues.

λ∗jkxg

Estimates EPC-interestEst. s.e. Task 1 Task 2 Task 3

Class 1 GDP -0.035 (0.007) 0.073 0.252 0.005Class 2 GDP -0.198 (0.012) -0.163 -0.058 0.002

Class 1 Women 0.013 (0.001) -0.003 0.029 0.002Class 2 Women -0.037 (0.001) -0.006 -0.013 0.002

Free ``loadings'' for task 1 and task 2.


References

Table : Partially invariant multilevel latent class model: EPC-interestvalues.

λ∗jkxg

Estimates EPC-interestEst. s.e. Task 1 Task 2 Task 3

Class 1 GDP -0.127 (0.008) 0.097Class 2 GDP 0.057 (0.011) 0.161

Class 1 Women 0.008 (0.001) 0.001Class 2 Women 0.020 (0.001) 0.007


References

WhathasbeengainedbyusingEPC-interest:

I am fairly confident here that there truly is "approximatemeasurement invariance", in the sense that any violations ofmeasurement invariance do not bias the primary conclusions.

I think attaining this goal is the main purpose of model fitevaluation.


References

Conclusion


References

Whatistheproblem?• We do latent variable modeling with a goal in mind.• But the latent variable model might be misspecified.• The appropriate question: "will that affect my goal?"• The actual question: "do the data fit the model in the

population" (LR) or "are the model and the data far apartrelative to model complexity" (RMSEA etc.)

Whatisthesolution?

Evaluatedirectlywhateffectpossiblemisspecificationshaveonthegoaloftheanalysis.


References

References

Oberski, D. (2014). Evaluating sensitivity of parameters of interest tomeasurement invariance in latent variable models. PoliticalAnalysis, 22(1):45--60.

Oberski, D. and Vermunt, J. (2013). A model-based approach togoodness-of-fit evaluation in item response theory. Measurement:InterdisciplinaryResearch&Perspectives, 11:117--122.

Oberski, D., Vermunt, J., and Moors, G. (frth). Evaluatingmeasurement invariance in categorical data latent variable modelswith the EPC-interest.

Vermunt, J. K. and Magidson, J. (2013). TechnicalguideforLatentGOLD 5.0: Basicandadvanced. Statistical Innovations Inc.,Belmont, MA.


SEM regression coefficient example

Two problems with invariance testing

Misspecified invariance model fit``Good'' fit ``Bad'' fit

ConclusionsUnaffectedby misspec-ification

(1) √ (2) Overparameteriza-tion or unnecessarilyd i s c a r d e d i t e m ,group, or scale.

Affected bymisspecifi-cation

(3) Non-invariance in-validates conclusions.

(4) √



European Sociological Review 2008, 24(5), 583--599Measurement invariance using the EPC-interest Daniel Oberski


Conservation Self−transcendence

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

SwedenDanmark

AustriaSwitzerlandNetherlands

GermanyIrelandSpain

NorwayHungaryFinland

PortugalFrance

BelgiumSlovenia

United KingdomGreece

Czech RepublicPoland

SwedenDanmark

AustriaSwitzerlandNetherlands

GermanyIrelandSpain

NorwayHungaryFinland

PortugalFrance

BelgiumSlovenia

United KingdomGreece

Czech RepublicPoland

ALLO

WN

OC

ON

D

−1.0 −0.5 0.0 0.5 1.0 −1.0 −0.5 0.0 0.5 1.0Regression coefficient



EPC-interest statistics of at least 0.1 in absolute value withrespect to the latent variable regression coefficients.

Metric invariance (loading) restriction``Conditions → Work skills'' in...

Slovenia France Hungary IrelandEPC-interest w.r.t.:Conditions →

Self-transcendence -0.073 -0.092 -0.067 0.073Conservation 0.144 0.139 0.123 -0.113

SEPC-self 0.610 0.692 0.759 -0.514



What has been gained by using EPC-interest

• Full metric invariance model: "close fit";• EPC-interest still detects threats to cross-country

comparisons of regression coefficients;

• MI and EPC-self do not detect these particularmisspecifications;

• MI and EPC-self detect other misspecifications;• Looking at EPC-interest reveals that these do not affect

the cross-country comparisons of regression coefficients.





comparisons of regression coefficients;• MI and EPC-self do not detect these particular

misspecifications;

• MI and EPC-self detect other misspecifications;• Looking at EPC-interest reveals that these do not affect







misspecifications;• MI and EPC-self detect other misspecifications;

• Looking at EPC-interest reveals that these do not affectthe cross-country comparisons of regression coefficients.






misspecifications;• MI and EPC-self detect other misspecifications;• Looking at EPC-interest reveals that these do not affect



ExampleGoal: Estimate gender differences in "valuing Stimulation":

(1) Very much like me; (2) Like me; (3) Somewhat like me; (4) Alittle like me; (5) Not like me; (6) Not like me at all.

impdiff (S)he looks for adventures and likes to take risks.(S)he wants to have an exciting life.

impadv (S)he likes surprises and is always looking for newthings to do. He thinks it is important to do lots ofdifferent things in life.

Tool: Structural Equation Model for European Social Surveydata (n = 18519 men and 16740 women).(OriginalstudybySchwarzetal. 2005)


Assume: Butreally(?):

What difference does it make for the goal: true genderdifferences in values? (re-analysisofdatabyOberski2014)

●

●●

●

●

●

●

●

Men value moreWomen value more

−0.2

0.0

0.2

ACPO ST SD HE COTR SE UN BE"Human value" factor

Late

nt m

ean

diffe

renc

e es

timat

e ±

2 s.

e.

Model

● Scalar invariance

Free intercept 'Adventure'


PROBLEM

The original authors found that the conditional independencemodel fit the data "approximately" (p. 1013)...

"Chi-squaredeterioratedsignificantly, ∆χ2(19) = 3313,p < .001, butCFI didnotchange. Changeinchi-squareishighlysensitivewithlargesamplesizesandcomplexmodels. Theotherindicessuggestedthatscalarinvariancemightbeaccepted(CFI =.88, RMSEA =.04, CI =.039.040,PCLOSE =1.0).''

... but unfortunately this "acceptable" misspecification couldreversetheirconclusions!


EPC-interest applied to Stimulation example

• After fitting the full scalar invariance model,• Effect size estimate of sex difference in Stimulation is

+0.214 (s.e. 0.0139).

• But EPC-interest of equal "Adventure" item intercept is-0.243.

• So EPC-interest suggests conclusioncanbereversedby freeing a misspecified scalar invariance restriction

• Actualchange when freeing this intercept is very close toEPC-interest: -0.235.


EPC-interest applied to Stimulation example

• After fitting the full scalar invariance model,• Effect size estimate of sex difference in Stimulation is

+0.214 (s.e. 0.0139).• But EPC-interest of equal "Adventure" item intercept is-0.243.

• So EPC-interest suggests conclusioncanbereversedby freeing a misspecified scalar invariance restriction

• Actualchange when freeing this intercept is very close toEPC-interest: -0.235.


evaluating sensitivity of parameters of interest to ... · references conclusion • this talk...

Documents