Assessing Hypothesis TestingFit Indices

Kline Chapter 8 (Stop at 210)Byrne page 68-84

Fit Indices

• It’s complicated yo.– There are a lot of them.– They do not imply that you are perfectly right.– They have guidelines but they are not perfect

either.– People misuse them.– Etc.

Fit Indices

• Limitations:– Fit statistics indicate an overall/average model fit. • That means there can be bad sections, but the overall

fit is good.

– No one magical number/summary.– They do not tell you where a misspecification

occurs.

Fit Indices

• Limitations– Do not tell you the predictive value of a model.– Do not tell you if it’s theoretically meaningful.

Fit Indices

• Model test statistic – examines if the reproduced correlation matrix matches the sample correlation matrix– Sometimes called “badness of fit”– Want these to be small

Fit Indices

• Traditional NHST = reject-support context– You reject the null to show your research

hypothesis is correct.• SEM Hyp Testing = accept-support context– You do not reject the null showing that your model

is consistent with the normal

Fit Indices

• Both types of statistical inferences have their problems … and especially in SEM because it is easy to find statistics that you would normally reject, even with good model fit.

• Tends to be too black and white (reject or not to reject!)

Fit Indices

• Approximate Fit Indices – Not traditionally a dichotomous yes-no decision – Do not distinguish between sampling error and

evidence against the model

Fit Indices

• Approximate Fit Indices– Absolute Fit Indices– Incremental Fit Indices– A parsimony-adjusted Index– Predictive Fit Indices

Fit Indices

• Absolute fit indices – Proportion of the covariance matrix explained by

the model– You can think about these as sort of R2

– Want these values be high

Fit Indices

• Incremental Fit Indices– Also known as comparative fit indices– Compared to the improvement over the

independence model (remember that’s the one with no relationships between the variables)

– Not necessarily the best indices

Fit Indices

• Parsimony-adjusted index– These include penalties for model complexity

(which normally gives you better fix by adding more paths)

– These will have smaller values for simpler models

Fit Indices

• Predictive fix indices– Estimate model fit in a hypothetical replication of

the study with the same sample size randomly drawn from the population

– Not always used

Fit Indices

• What size? I need a rule?!– Everyone cites Hu and Bentler (1999) for the

golden standards.– Same problem that Cohen had (we love rules).

• So when the fit is messy, cite Kline (page 197) as reasons that’s not a bad thing– This section is an interesting read, especially if you

have trouble publishing, but not crucial to your understanding of fit indices

Model Test Statistic

• Chi-square (listed as CMIN in your output)– Formula = (N-1)FML

– FML = is the minimum fit function in ML estimation– P values are based on df for your model and a chi-

square distribution– You want this to be nonsignificant.• But this is a catch 22!

Model Test Statistic

• Chi-square is biased by– Multivariate non-normality – Correlation size – bigger correlations can be bad

for you (harder to estimate all that variance)– Unique variance – Sample size

Model Test Statistic

• Everyone reports chi-square, but people tend to ignore significant values– (I’m sort of eh on his YOU MUST PAY ATTN OR DIE

talk in this section)

Model Test Statistic

• Normed chi-square or (X2/df) – this used to be widely reported and used– The criterion was < 3.00 were good models– Now most people have moved away from this

procedure

Approximate Fit Indices

• Some examples:– RMSEA (root mean square error of approximation)– SRMR (standardized root mean square residual)

– A/GFI (adjusted/goodness of fit index) – CFI (comparative fix index)– TLI (Tucker-Lewis Index)– NFI (Normed Fit Index)

RMSEA

• Parsimony-adjusted index• Want small values– Excellent < .06 (not a typo different than book)– Good < .08– Acceptable < .10– Eeek > .10

• Report CI!

Pclose

• Tests if the RMSEA is in the excellent range• You want p > .50 to show that there is a high

probability that RMSEA is effectively zero

SRMR

• Parsimony-adjusted index• Want small values– Excellent < .06 (not a typo different than book)– Good < .08– Acceptable < .10– Eeek > .10

GFI

• Do not use this sucker unless you want to get a nasty review.– GFI, AGFI, PGFI

• Lots of research showing it’s positively biased• Want large values

CFI

• Incremental Fit Index– Values are 0 to 1 (sometimes you’ll get slightly

over 1, usually indicates something is wrong)– Want high values• Excellent >.95• Good > .90• Blah < .90

The Other FIs

• NFI – a variation of the CFI, as it was said to underestimate for small samples

• RFI (relative fit index)• IFI (incremental fit index)• TLI (Tucker Lewis Index)– All have the same basic rules and formulas as CFI

See Tabachnick 720-725 for how these and the next slides are calculated

Some other statistics

• Pratio – parsimony index• PNFI, PCFI are parsimony adjustments for NFI,

CFI• NCP – noncentrality parameter (tells you how

much it leans from the normal for that distribution)

Some other statistics

• FMIN – minimum discrepancy function used to calculate chi-square and other statistics – Include confidence interval in Amos

Model Comparisons

• Let’s say you want to adjust your model– You can compare the adjusted model to the

original model to determine if the adjustment is better

• Let’s say you want to compare two different models– You can compare their fits to see which is better

Model Comparisons

• Nested models– If you can create one model from another by the

addition or subtraction of parameters, then it is nested• Model A is said to be nested within Model B, if

Model B is a more complicated version of Model A. – For example, a one-factor model is nested within a

two-factor as a one-factor model can be viewed as a two-factor model in which the correlation between factors is perfect).

Nested Models

• Chi-square difference test– | Subtract Model 1 CMIN – Model 2 CMIN |– Subtract Model 1 df – Model 2 df– Use a chi-square table to look up p < .05 for

difference in df– See if the first step is greater than that value• If yes, you say the model with the lower chi-square is

better• If no, you say they are the same and go with the

simpler model

Nested Models

• So how can I tell what to change?• NOTE: JUST CHANGE ONE THING AT A TIME!• Use modification indices!– They tell you what the chi-square change would be

if you add the path suggested.– Based on X2(1) – called a Lagrange Multiplier

• Remember that p < .05 = 3.84, so Amos automatically gives you everything > 4.

• You can change this to see fewer options if you have a lot.

Non-Nested Models

• AIC – Akaike Information Criterion– Related CAIC (consistent AIC)

• BIC – Bayesian Information Criterion• BCC – Browne-Cudeck Criterion

– All of these are how much the sample will cross validate in the future

– You want them to be small, so you pick the smallest one of the two models (how different?)

Non-Nested Models

• ECVI – expected cross validation index• MECVI – modified ECVI

– Again, you want small values, so you pick the model with the smallest ECVI

OMG!

• So what to do?– Mainly people report: X2, RMSEA, SRMR, CFI– Determine the type of model change to use the

right model comparison statistic