Testing factorial invariance in multilevel data: A Monte Carlo study

Eun Sook Kim

Oi-man Kwok

Myeongsun Yoon

The purpose of the present study

Investigate the influence of data dependency on the test of factorial invariance in multilevel data. Comparisons between multilevel SEM (the model-

based or the designed-based multiple-group multilevel CFA) and multiple-group CFA

Measurement invariance studies in multilevel data

Multilevel SEM The groups of interest are treated as random samples. Good for measurement invariance testing in individual

and organizational levels. Multiple-group CFA

The number of groups is small or relatively finite (e.g., <20 or so).

Can be utilized to test measurement invariance over groups at the organizational level.

Multilevel measurement model

A conventional measurement model

A multilevel measurement model

In a multilevel measurement model

The variance of the factor

The variance of residual

The variance of observed scores

Examine measurement invariance in multilevel data

The general multilevel CFA model

Examine measurement invariance in multilevel data

Test at the between level

Test at the within level

Study 1

Study 2

Data generation

Design

Noninvariance in the organizational (study 1) and the individual level (study 2).

Manipulated factors The number of clusters (CN):30, 50, 80 and 160 Cluster size (CS): 10 and 20 The size of ICC: .09, .20, and .33

The total sample size was between 300 and 3,200, and 1,000 replications were in a condition.

Data dependency on group membership:

Intraclass correlation (ICC)

Analysis

Tested a configural and a metric models. The LRT was used to test factorial invariance

in multiple-group CFA . The Satorra-Bentler scaled chi-squared test

was used in multiple-group multilevel CFA. Examined power and type I error. Conducted ANOVA to examine the influence

of 3 manipulated factors on power and type I error.

Results (Study 1)Low admissible rates.

Analysis in study 2

Examined factorial invariance at the individual level using the design-based multilevel CFA, instead of the model-based one, due to the limitation of the SEM software.

Results (study 2)

Limitations and conclusions

The explanations of the discrepant findings between study 1 and study 2 might be confounded by the type of analysis.

It is necessary to take account of the data dependency in multilevel data.

A substantially large number of clusters is required in model-based multiple-group multilevel modeling.

Comments

Information of the accuracy of estimated parameters is not available. MLR provides accurate factor loadings but

underestimates the residual variances and standard errors at the between level. Having more clusters or higher ICC does not effectively compensate for this.

Groups of interest might occur at different levels simultaneously, and how to model them?

Comments (Cont.)

It is not clear how the model was constrained. This study seemed to constrain the variance of the latent factor to be equal across groups, which is not practical and reasonable in real data.

The large effect size in the factorial invariance with a large sample size yielded very high power, and it is recommended reducing the effect size to medium.

Comments (Cont.)

Contrast to the findings of Jones-Farmer (2010), the inflated type I error occurred in CFA even though data dependency was considered small. Further examinations are needed to understand the controversial findings.

Future studies Further examinations in

Scalar and error variance models Cross-level invariance

The issue of sample sizes using other estimation methods In MLR, the min. sample size of the between level

is 100. How about in FIML? The assessment of model fit in ML SEM.

Future studies (Cont.) Examine more than 1 items with different

factor loadings. Implement purification in this study. Conduct equal means for the latent factor or

equal intercepts of indicators to examine factorial invariance.

Examine scalar invariance to see if the findings are similar to the findings of French and Finch (2010).

Future studies (Cont.)

The possibility to develop a new indicator in SEM, such as adding person fit (as IRT approaches do) to SEM approaches, instead of model fit only.

Conduct the model-based multilevel CFA using other software to overcome the limitation in current SEM software.