generalizability theory nothing more practical than a good theory!
TRANSCRIPT
Generalizability Theory
Nothing more practical than a good theory!
This presentation is made by Prof. ZhaoThis presentation is made by Prof. Zhao
Overview of Presentation
Classes of reliability theories Generalizability Theory
G-study D-study
Illustrations
Three Reliability Theories
Classical Test Theory Generalizability Theory Item Response Theory
Overview of Presentation
Classes of reliability theories Generalizability Theory
G-study D-study
Illustrations
Generalizability Theory
Fundamental is the concept of parallel measures (like classical test theory), but the theory allows a multitude of error sources
Generalizability concept:Reliability is dependent on the inferences (generalizations) that the investigator wishes to make with the data from the measurement
Illustration
Essay test 7 vignette based essay questions 2 markers independently marking all
questions for all examinees Reliability in a classical framework:
Cronbach’s alpha: 0.66 Inter rater reliability (i.e. kappa) 0.71
Fundamental Equation
X =X = Observed score
T + E T = True score
E = Error score
Reliability = Variance of TVariance of X
The larger the variance of T in relation to X, the higher the
reliability
Fundamental Equation
X =X = Observed score
T + E T = True score
E = Error score
Reliability = Variance of TVariance of X
= = =
Fundamental Equation
X =X = Observed score
T + E T = True score
E = Error score
Reliability = Variance of TVariance of X
Reliability = Variance of TVar T + Var E
Multiple sources of error variance
Reliability = Variance of TVar T + Var E
Markers Essays Unexplained
Two steps in G analysis
1) G(eneralizability)-study:Estimation of sources of variance that influence the measurement (e.g., variance between examinees, essays and markers)
2) D(ecision)-study:Estimation of reliability indices as a function of concrete sample size(s) (e.g., number of essays, number of markers)
G-study steps
Determine facets (factors of variance)
Determine design Random vs fixed Crossed vs nested
Crossed vs nested designs
A B
1
2
3
4
5
6
A B C D E F G H I J K L
Crosseddesign
Nesteddesign
G-study
Determine facets (factors of variance)
Determine design Random vs Fixed Crossed vs nested Collect data
Analysis of Variance (ANOVA) Estimation of variance components
Illustration 1
Essay Test 7 vignette based open ended questions 100 students One marker marked all essays for all
students G-study questions?
N of factors/facets? Random/fixed facets? Nested or crossed?
One facet designRandomCrossed
Sources of Variance
Person x Items
p ipi,e
Sources of Variance
Person x Items
ip pi,e
Sources of Variance
Person x Items
p ipi,e
Sources of Variance
Person x Items
p pi,e
Variance component estimation (one facet design)
An observed score for a person on an item (Xpi):
Xpi = [Overall mean]
+ p - [Person effect]
+ i - [Item effect]
+ pi - p - i - [Residual]
Each of these effects have an average (always 0) anda variance (2). The latter ones are the variance components.
The variance of all observes scores Xpi across all persons and items:
^
^2 (Xpi) = ^2p
^2i + ^2
pi,e +
Variance components
P x I design
Source
pi
pi,e
EstimatedVariance
Component
97.57261.24371.97
StandardError
19.02112.9817.60
Percentageof TotalVariance
13.3535.7550.90
Crossed vs nested designs
A B
1
2
3
4
5
6
A B C D E F G H I J K L
Crosseddesign
Nesteddesign
Sources of Variance
Items : Persons
p i,pi,e
Variance components
I : P design
p
i,pi,e
97.57
663.21
35.7550.90
13.35
86.65
ipi,e
261.24371.97
Source
EstimatedVariance
Component
Percentageof TotalVariance
Variance components
I : P design
p
i,pi,e
97.57
663.21
35.7550.90
13.35
86.65
ipi,e
261.24371.97
Source
EstimatedVariance
Component
Percentageof TotalVariance
pi,pi,e
97.57663.21
13.3586.65
Sources of Variance
Person x Items x Judges
p i
pij,e
pi
pj ij
j
Variance component estimation (two facet design)
An observed score for a person on an item (Xpi):
Xpi = [Overall mean]
+ p - [Person effect]
+ j - [Item effect]
+ i - [Judge effect]
+ pj - p - j + [Person by judge effect] + pi - p - i + [Person by item effect]
+ ij - j - i + [Judge x item effect]
+ pij - pj - pi - ij + p + j + i - [Residual]
The variance of observes scores Xpi across all persons and items:
^2 (Xpij) = ^2p
^2j + + ^2
i + ^2pj +
^2pi +
^2ij +
^2pij,e
Variance componentsP x I x J design
Source
pij
pipjij
pij,e
EstimatedVariance
Component
48.7125.1215.00
185.8733.1880.0072.94
Percentageof TotalVariance
10.575.453.26
40.337.20
17.3615.83
Overview of Presentation
Classes of reliability theories Generalizability Theory
G-study D-study
Illustrations
Two steps in G analysis
1) G(eneralizability)-study:Estimation of sources of variance that influence the measurement (e.g., variance between examinees, essays and markers)
2) D(ecision)-study:Estimation of reliability indices as a function of concrete sample size(s) (e.g., number of essays, number of markers)
Interpretation of scores
Norm-oriented perspectiveScores have relative meaning; scores have meaning in relation to each other
Domain-oriented perspectiveScores have absolute meaning to the domain of measurement
Mastery-oriented perspectiveScores have meaning in relation to a cut-off score (reliability of decisions, not of scores)
Fundamental Equation
X =X = Observed score
T + E T = True score
E = Error score
Reliability = Variance of TVariance of X
Reliability = Variance of TVar T + Var E
Illustration 1
Essay test 7 vignette based essay questions 1 markers marked all questions for all
examinees Norm-referenced perspective
Calculate generalizability coefficient!
D-study (ni = 7; norm-referenced)
Source
pi
pi,e
EstimatedVariance
Component
97.57261.24371.97
StandardError
19.02112.9817.60
Percentageof TotalVariance
13.3535.7550.90
G =T
T + E=
97.57
97.57 + 371.97/7= 0.65
Illustration 2
Essay test 7 vignette based essay questions 1 markers marked all questions for all
examinees Domain-referenced perspective
Calculate dependability coefficient!
D-study (ni = 7; domain referenced)
Source
pi
pi,e
EstimatedVariance
Component
97.57261.24371.97
StandardError
19.02112.9817.60
Percentageof TotalVariance
13.3535.7550.90
D =97.57
97.57+= 0.52
261.24/ 7
+371.97/ 7
Illustration 3
Essay test 7 vignette based essay questions 1 markers marked all questions for all
examinees Domain-referenced perspective
Calculate dependability coefficient fora sample of 10 essays!
D-study (ni = 10; domain referenced)
Source
pi
pi,e
EstimatedVariance
Component
97.57261.24371.97
StandardError
19.02112.9817.60
Percentageof TotalVariance
13.3535.7550.90
D =97.57
97.57+= 0.61
261.24/10
+371.97/ 10
D-studies for several item samples
N Essays
1571015
GeneralizabilityCoefficient (G)
0.210.570.650.720.80
DependabilityCoefficient (D)
0.130.440.520.610.70
Illustration 4
Essay test 7 vignette based essay questions 2 markers independently marked all
questions for all examinees Norm-referenced perspective
Calculate generalizability coefficient!
D-study (ni=7; nj=2; norm referenced)
Source
pij
pipjij
pij,e
VarianceComponent
48.7125.1215.00
185.8733.1880.0072.94
% of TotalVariance
10.575.453.2640.337.2017.3615.83
G =48.71
48.71+= 0.50
185.87/ 7
+33.18/2
+72.94/2 x 7
Illustration 5
Essay test 7 vignette based essay questions 2 markers independently marked all
questions for all examinees Domain-referenced perspective
Calculate dependability coefficient!
D-study (ni=7; nj=2; domain referenced)
Source
pij
pipjij
pij,e
VarianceComponent
48.7125.1215.00
185.8733.1880.0072.94
% of TotalVariance
10.575.453.2640.337.2017.3615.83
D =48.71
48.71+= 0.43
25.12/ 7
+15.00/2+185.87/
14+33.18/
2+80.00/
14+72.94/
14
Illustration 6
Essay test 7 vignette based essay questions 2 different markers
independently marked each question for all examinees
Norm-referenced perspective
Calculate generalizability coefficient!
D-study (ni=7; nj=2; norm referenced)
SourceEstimated Var
ComponentPerc of Total
Variance
(Judges : Items) x Persons
pi
j,ijpi
pj,pij,e
48.7125.1895.00
185.87106.12
10.575.45
20.6240.3323.03
G =48.71
48.71+= 0.52
185.87/ 7
+ 106.12/2 x 7
D-study summary table
TwoMarkers
0.440.500.560.61
OneMarker
0.390.470.560.65
TwoMarkers
0.460.540.630.72
Same Markerfor all essays
Different Markerfor each essayNumber
ofEssays
571015
OneMarker
0.360.410.450.49
Norm-referenced score interpretation
Another reliability index
Reliability coefficient (G & D coefficients) Scale independent (0-1) Non-intuitive interpretation
Standard Error of Measurement (SEM) Intuitive interpretation Scale dependent
Standard Error of Measurement
X =X = Observed score
T + E T = True score
E = Error score
Reliability index = Variance of TVariance T + Variance E
EStandard Error of Measurement (SEM) =
Interpretation of SEM
Suppose an examinee has a score of 60% and the SEM is 5:
60555045 65 70 7565% CI
1.96 x 5 10
60555045 65 70 7595% CI
2.14 x 5 11
60555045 65 70 7595% CI
D-study (ni = 7; norm referenced)
Source
pi
pi,e
EstimatedVariance
Component
97.57261.24371.97
StandardError
19.02112.9817.60
Percentageof TotalVariance
13.3535.7550.90
G =97.57
97.57 + 371.97/7= 0.65
SEM = = 7.29 371.97 /7
D-study (ni=7; nj=2; domain referenced)
Source
pij
pipjij
pij,e
VarianceComponent
48.7125.1215.00
185.8733.1880.0072.94
% of TotalVariance
10.575.453.2640.337.2017.3615.83
D =48.71
48.71+= 0.43
25.12/ 2
+15.00/2+185.87/
14+33.18/
2+80.00/
14+72.94/
14SEM = = 8.57
Overview of Presentation
Classes of reliability theories Generalizability Theory
G-study D-study
Illustrations
Scenario CEX
A clinical mini exercise (CEX) was developed in which examinees are periodically observed and rated on a rating form. An investigator analyzed a data set from 88 residents who were each observed on 4 occasions by a single different examiner (cf. 1. Norcini JJ, Blank LL, Arnold GK, Kimbal HR. The mini-CEX (Clinical Evaluation Exercise): A
preliminary investigation. Annals of Internal Medicine 1995;123:795-799.). Variance
Componentsp
o,op,eG =
p
p + o:p /4
= Do:p
Scenario OSCE I
An OSCE was administered to 100 final year students consisting of 15 stations. Each station was scored by two independent examiners on a case specific checklist. Different examiners were used in each station.
VarianceComponents
ps
G =p
p +j:spspj:s
ps /15
+ pj:s /2 x15
Scenario OSCE II
An experimental OSCE was administered to 20 residents. Each resident was tested on a different day. For each resident 3 stations were organized consisting of real patients that were available that day. Two examiners observed all residents in all stations and completed a generic rating scale.
VarianceComponents
ps:p D =
p
p +s:p /3j
ps:spj
+ j /2+ ps:s /
3+ pj /
6
Scenario Clerkship Evaluation
An investigator wishes to evaluate teaching quality of 10 clinical clerkships. She developed a questionnaire with 30 items on various quality aspects. The questionnaire was administered in all clerkships by 50 students.
VarianceComponents
ci
s:cci
cs:i
G =c
c + s:c /50
+ ci /30
+ cs:i /50 x 30
PS: It is doubtful that i is a random facet and i could be treated as fixed or ignored!
Further reading & software
Literature Cronbach LJ, Gleser GC, Nanda H, Rajaratnam N. The dependability of behavioral
measurements: Theory of generalizability for scores and profiles. New York: Wiley, 1972. Original monograph on generalizability theory. Complete, but hardly accessible for any reader.
Brennan RL. Elements of Generalizability Theory. Iowa: ACT Publications, 1983.This is the resource book for most specialists. Not easy for non-statistically trained readers
Shavelson RJ, Webb NM. Generalizability theory: A primer. Newbury Park, CA: Sage Publications, 1991 . Good and accessible introduction to generalizability theory for any reader
Software GENOVA
Conducts G and D studies and provides ample statistical information. Operates on any PC. Program is relatively old and not user friendly. Program available from Dr. J. Crick, National Board of Medical Examiners, National Board of Medical Examiners, 3750 Market Street,Philadelphia, PA 19104-3190, USA.
SPSSSPSS General Linear Models, Subprogram Variance Components, estimates variance components (also for unbalanced designs). D-studies need to be done manually.