the theory of sampling and measurement. sampling first step in implementing any research design is...

The Theory of The Theory of Sampling and Sampling and MeasurementMeasurement

SamplingSampling

First step in implementing any First step in implementing any research design is to create a sample. research design is to create a sample.

We cannot study the theoretical We cannot study the theoretical population of all conceivable events population of all conceivable events (e.g., events that have not occurred), (e.g., events that have not occurred), nor can we usually study all instances nor can we usually study all instances of actual events. We select some of actual events. We select some instances to study and not others. instances to study and not others. Those we include are our Those we include are our samplesample. .

How our sample is selected is critical How our sample is selected is critical for external validity or generalizability.for external validity or generalizability.

Who do you want Who do you want to generalize to?to generalize to?Who do you want Who do you want to generalize to?to generalize to?

Groups in SamplingGroups in SamplingGroups in SamplingGroups in Sampling

Groups in SamplingGroups in SamplingGroups in SamplingGroups in SamplingThe theoretical The theoretical

populationpopulationThe theoretical The theoretical

populationpopulation

What population can What population can you get access to?you get access to?

What population can What population can you get access to?you get access to?




Groups in SamplingGroups in SamplingGroups in SamplingGroups in SamplingThe Theoretical The Theoretical

PopulationPopulationThe Theoretical The Theoretical

PopulationPopulation

The study The study populationpopulationThe study The study populationpopulation

How can you get How can you get access to them?access to them?How can you get How can you get access to them?access to them?









The sampling The sampling frameframe


Who is in your study?Who is in your study?Who is in your study?Who is in your study?













The sampleThe sampleThe sampleThe sample

Types of SamplesTypes of Samples

Probability Probability SamplingSampling Simple randomSimple random Stratified Stratified

randomrandom Cluster or area Cluster or area

randomrandom

Non-Probability Non-Probability SamplingSampling AccidentalAccidental Modal instanceModal instance ExpertExpert SnowballSnowball Case study Case study

(intentional (intentional selection)selection)

The Sampling The Sampling DistributionDistribution

The Sampling The Sampling DistributionDistribution

AverageAverageAverageAverage AverageAverageAverageAverage AverageAverageAverageAverage

4.44.24.03.83.63.43.23.0

15

10

5

0

The sampling The sampling distribution...distribution...The sampling The sampling distribution...distribution...

...is the distribution ...is the distribution of a statistic across of a statistic across an infinite number an infinite number

of samples.of samples.

SampleSample

4.44.24.03.83.63.43.23.0

5

0

5

0

SampleSample

4.44.24.03.83.63.43.23.0

5

0

5

0

SampleSample

4.44.24.03.83.63.43.23.0

5

0

5

0

Population ParameterPopulation ParameterPopulation ParameterPopulation Parameter

4.54.03.53.0

150

100

50

0

Self esteemSelf esteem

Fre

qu

ency

Fre

qu

ency

The population has The population has a mean of 3.75...a mean of 3.75...


...and a ...and a standard unit standard unit

of .25.of .25.

...and a ...and a standard unit standard unit

of .25.of .25.

This meansThis meansAbout 64% of cases fall between 3.5 - 4.0.About 64% of cases fall between 3.5 - 4.0.

About 95% of cases fall between 3.25 - 4.25.About 95% of cases fall between 3.25 - 4.25.

about 99% of cases fall between 3.0 - 4.5about 99% of cases fall between 3.0 - 4.5

Sampling DistributionSampling DistributionSampling DistributionSampling Distribution

4.54.03.53.0

150

100

50

0

Self-esteemSelf-esteem

Fre

qu

ency

Fre

qu

ency

The population has The population has a mean of 3.75.a mean of 3.75.

The population has The population has a mean of 3.75.a mean of 3.75.

Sampling DistributionSampling DistributionSampling DistributionSampling Distribution

4.54.03.53.0

150

100

50

0

Self-esteemSelf-esteem

Fre

qu

ency

Fre

qu

ency



...and a ...and a standard standard

error of .25.error of .25.

...and a ...and a standard standard

error of .25.error of .25.

Inferring Population Inferring Population from Samplefrom Sample

Inferring Population Inferring Population from Samplefrom Sample

4.54.03.53.0

150

100

50

0

Self esteemSelf esteem

Fre

qu

ency

Fre

qu

ency

The sample has a The sample has a mean of 3.75...mean of 3.75...

The sample has a The sample has a mean of 3.75...mean of 3.75...

...and a ...and a standard standard deviation deviation

of .25.of .25.

...and a ...and a standard standard deviation deviation

of .25.of .25.

This meansThis means64% chance true population mean falls between 3.5 - 4.0.64% chance true population mean falls between 3.5 - 4.0.

95% chance true population mean falls between 3.25 - 4.25.95% chance true population mean falls between 3.25 - 4.25.

99% chance true population mean falls between 3.0 - 4.599% chance true population mean falls between 3.0 - 4.5

Figure 3.4 Labor Repression and Growth in Figure 3.4 Labor Repression and Growth in the Asian Cases, 1970-1981the Asian Cases, 1970-1981

Philippines

Thailand

Indonesia

Singapore

Taiw anMalaysiaKorea

-4

-2

0

2

4

6

8

10

0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0

Labor Repression

Growth in GDP per capita

Figure 3.5 Labor Repression and Growth in the Figure 3.5 Labor Repression and Growth in the Full Universe of Developing Countries,1970-1981Full Universe of Developing Countries,1970-1981

Israel Philippines

Syria

Iran

Korea Taiw an

BrazilMexico

Malaysia

Jamaica

Uganda

Botsw ana

Singapore

-4

-2

0

2

4

6

8

10

0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0

Labor Repression

Growth in GDP per capita

MeasurementMeasurement

Operationalization is the process of Operationalization is the process of translating theoretical constructs into translating theoretical constructs into observable indicators.observable indicators.

Construct validity and reliability are the Construct validity and reliability are the criteria we use to evaluate how well you criteria we use to evaluate how well you have operationalized your concepts.have operationalized your concepts.

Both matter regardless of the level of Both matter regardless of the level of measurement and whether you are using measurement and whether you are using qualitative or quantitative indicators.qualitative or quantitative indicators.

The Hierarchy of LevelsThe Hierarchy of Levels

NominalNominalNominalNominal

IntervalIntervalIntervalInterval

RatioRatioRatioRatio

Attributes are only named; weakest

Attributes can be ordered

Distance is meaningful

Absolute zero

OrdinalOrdinalOrdinalOrdinal

Nominal MeasurementNominal Measurement The values “The values “namename” the attribute ” the attribute

uniquely.uniquely. The name does The name does notnot imply any imply any

ordering of the cases.ordering of the cases.

Ordinal MeasurementOrdinal MeasurementWhen attributes can be When attributes can be rank-rank-

orderedordered…… Distances between attributes Distances between attributes do do

not have any meaningnot have any meaning..

Interval MeasurementInterval MeasurementWhen When distancedistance between attributes has between attributes has

meaning, for example, temperature meaning, for example, temperature (in Fahrenheit) -- distance from 30-(in Fahrenheit) -- distance from 30-40°F is same as distance from 70-40°F is same as distance from 70-80°F80°F

Note that Note that ratios don’t make any ratios don’t make any sense -sense -- 80°F is not - 80°F is not twicetwice as hot as as hot as 40°F.40°F.

Ratio MeasurementRatio Measurement Has an Has an absolute zero absolute zero that is that is

meaningfulmeaningful Can construct a meaningful Can construct a meaningful ratioratio

(fraction), for example, number of (fraction), for example, number of clients in past six monthsclients in past six months

Construct ValidityConstruct Validity

Key problem is that we have abstract Key problem is that we have abstract theoretical construct – power, democracy, theoretical construct – power, democracy, development, corruption, etc. – that we development, corruption, etc. – that we can never observe directly.can never observe directly.

Yet, to test propositions requires that we Yet, to test propositions requires that we have some indicator for the construct – or have some indicator for the construct – or at least have proxies that we can argue at least have proxies that we can argue are capturing some attributes of the are capturing some attributes of the construct.construct.

Our indicator is an analogy (to an Our indicator is an analogy (to an analogy).analogy).

Assessing Construct Assessing Construct ValidityValidity

Translation ValidityTranslation Validity Face Validity: plausible on its “face”Face Validity: plausible on its “face” Content Validity: matches lists of Content Validity: matches lists of

attributesattributes Criterion-related ValidityCriterion-related Validity

Predictive Validity: predicts accuratelyPredictive Validity: predicts accurately Concurrent Validity: distinguishes Concurrent Validity: distinguishes

appropriately between groupsappropriately between groups Convergent ValidityConvergent Validity Discriminant ValidityDiscriminant Validity

The Convergent PrincipleThe Convergent PrincipleThe Convergent PrincipleThe Convergent Principle

AlternativeAlternative measures of a measures of a construct construct shouldshould be strongly be strongly

correlated.correlated.

How It WorksHow It WorksHow It WorksHow It WorksTheoryTheoryTheoryTheory

Self-esteemSelf-esteemconstructconstruct

Item 1Item 1 Item 2Item 2 Item 3Item 3 Item 4Item 4

You You theorizetheorize that the that the items all reflect self-esteem.items all reflect self-esteem.

You You theorizetheorize that the that the items all reflect self-esteem.items all reflect self-esteem.


ObservationObservationObservationObservation


Item 1Item 1 Item 2Item 2 Item 3Item 3 Item 4Item 4

1.001.00 .83.83 .89.89 .91.91.83.83 1.001.00 .85.85 .90.90.89.89 .85.85 1.001.00 .86.86.91.91 .90.90 .86.86 1.001.00

The correlations provide evidence The correlations provide evidence that the items all that the items all convergeconverge

on the same construct.on the same construct.

The correlations provide evidence The correlations provide evidence that the items all that the items all convergeconverge

on the same construct.on the same construct.

Convergent Validity in Convergent Validity in Measures of “Democracy”Measures of “Democracy”

19851985 | polity2 pollib civlib | polity2 pollib civlib regreg

-------------+-------------------------------------------------+------------------------------------

polity2 | 1.0000 -0.9148 -0.8770 -0.8601 polity2 | 1.0000 -0.9148 -0.8770 -0.8601

pollib | -0.9148 1.0000 0.9176 0.8440 pollib | -0.9148 1.0000 0.9176 0.8440

civlib | -0.8770 0.9176 1.0000 0.8053 civlib | -0.8770 0.9176 1.0000 0.8053

reg | -0.8601 0.8440 0.8053 1.0000reg | -0.8601 0.8440 0.8053 1.0000

Convergent Validity in Convergent Validity in Measures of “Education”Measures of “Education”

19851985 | 1| 1 22 33 44 55 66-------------+-------------------------------------------------------------------+------------------------------------------------------Ed. spending Ed. spending | 1.0000 -0.1217 0.2415 0.3563 0.0214 | 1.0000 -0.1217 0.2415 0.3563 0.0214

0.01950.0195Illiteracy (%) Illiteracy (%) | -0.1217 1.0000 -0.5797 -0.7306 -0.8569 - | -0.1217 1.0000 -0.5797 -0.7306 -0.8569 -

0.61960.6196Cohort to Grade 4 | 0.2415 -0.5797 1.0000 0.4419 0.6553 0.3654Cohort to Grade 4 | 0.2415 -0.5797 1.0000 0.4419 0.6553 0.3654% Grade School % Grade School | 0.3563 -0.7306 0.4419 1.0000 0.6230 | 0.3563 -0.7306 0.4419 1.0000 0.6230

0.36120.3612% Secondary School | 0.0214 -0.8569 0.6553 0.6230 1.0000 0.7576% Secondary School | 0.0214 -0.8569 0.6553 0.6230 1.0000 0.7576% College % College | 0.0195 -0.6196 0.3654 0.3612 0.7576 1.0000 | 0.0195 -0.6196 0.3654 0.3612 0.7576 1.0000

The Discriminant The Discriminant PrinciplePrinciple

The Discriminant The Discriminant PrinciplePrinciple

Measures of Measures of differentdifferent constructs should constructs should notnot

correlate highly with each correlate highly with each other.other.



SESE11 SESE22

Locus-of-controlLocus-of-controlconstructconstruct

LOCLOC11 LOCLOC22


Self- esteemSelf- esteemconstructconstruct

SESE11 SESE22


LOCLOC11 LOCLOC22

You theorize that you have two distinguishable constructs.



SESE11 SESE22


LOCLOC11 LOCLOC22


rrSESE11, , LOCLOC11

= .12 = .12


= .09 = .09


= .04 = .04


= .11 = .11

The correlations provide evidence that the items on the two tests discriminate.

TheoryTheoryTheoryTheory


SESE11 SESE22 SESE33


LOCLOC11 LOCLOC22 LOCLOC33

We have two constructs. We have two constructs. We want to measure We want to measure self-esteemself-esteem

and and locus of control.locus of control.

For each construct, we develop threeFor each construct, we develop threescale items; our theory is that itemsscale items; our theory is that items

within the construct will converge andwithin the construct will converge andItems across constructs will discriminate.Items across constructs will discriminate.



Self-esteemSelf-esteemConstructConstruct




1.001.00 .83.83 .89.89 .02.02 .12.12 .09.09.83.83 1.001.00 .85.85 .05.05 .11.11 .03.03.89.89 .85.85 1.001.00 .04.04 .00.00 .06.06.02.02 .05.05 .04.04 1.001.00 .84.84 .93.93.12.12 .11.11 .00.00 .84.84 1.001.00 .91.91.09.09 .03.03 .06.06 .93.93 .91.91 1.001.00

SESE11

SESE22

SESE33

LOCLOC11

LOCLOC22

LOCLOC33

SESE11 SESE22 SESE33 LOCLOC11 LOCLOC22 LOCLOC33

GreenGreen and and redredcorrelations arecorrelations are

Convergent;Convergent;yellow areyellow are

Discriminant.Discriminant.







1.001.00 .83.83 .89.89 .02.02 .12.12 .09.09.83.83 1.001.00 .85.85 .05.05 .11.11 .03.03.89.89 .85.85 1.001.00 .04.04 .00.00 .06.06.02.02 .05.05 .04.04 1.001.00 .84.84 .93.93.12.12 .11.11 .00.00 .84.84 1.001.00 .91.91.09.09 .03.03 .06.06 .93.93 .91.91 1.001.00

SESE11

SESE22

SESE33

LOCLOC11

LOCLOC22

LOCLOC33

SESE11 SESE22 SESE33 LOCLOC11 LOCLOC22 LOCLOC33

The correlations support bothThe correlations support bothconvergenceconvergence and and discriminationdiscrimination,,and therefore and therefore constructconstruct validity. validity.

What Is Reliability?What Is Reliability?What Is Reliability?What Is Reliability?

The “repeatability” of a measureThe “repeatability” of a measure The “consistency” of a measureThe “consistency” of a measure The “dependability” of a measureThe “dependability” of a measure

True Score TheoryTrue Score Theory

12

34

5

12

34

5

3Scan a multitu

de of inform

ation and decide

what is im

portant.

12

34

5

12

34

5

12

34

5

12

34

5

12

34

5

1Manage tim

e effectively

2Manage resources effectively.

3Scan a multitu

de of information and

decide what is im

portant.

4Decide how to manage multip

le tasks.

5Organize the work when directions are not specific

.

1Manage tim

e effectively

Rating Sheet

ObservedObservedscorescore

ObservedObservedscorescore ==

TrueTrueabilityabilityTrueTrue

abilityability++ RandomRandom

errorerrorRandomRandom

errorerror

TT ee++XX

The Error ComponentThe Error Component

TT ee++XX

Two components:Two components:

• Random errorRandom error• Random errorRandom error

• Systematic errorSystematic error• Systematic errorSystematic error

eerr

eess

The Revised True Score The Revised True Score ModelModel

TT eerr++XX eess++

Random ErrorRandom Error

XX

Fre

qu

ency

Fre

qu

ency

The distribution of X The distribution of X with no random errorwith no random errorThe distribution of X The distribution of X with no random errorwith no random error

Random ErrorRandom Error

XX

Fre

qu

ency

Fre

qu

ency

The distribution of X The distribution of X with no random errorwith no random errorThe distribution of X The distribution of X with no random errorwith no random error

The distribution of X The distribution of X with random errorwith random errorThe distribution of X The distribution of X with random errorwith random error

Notice that random error doesn’t Notice that random error doesn’t affect the average, only the affect the average, only the

variability variability around the average.around the average.

Systematic ErrorSystematic Error

XX

Fre

qu

ency

Fre

qu

ency

The distribution of X The distribution of X with no systematic errorwith no systematic errorThe distribution of X The distribution of X with no systematic errorwith no systematic error

Systematic ErrorSystematic Error

XX

Fre

qu

ency

Fre

qu

ency

The distribution of X The distribution of X with no systematic errorwith no systematic errorThe distribution of X The distribution of X with no systematic errorwith no systematic error

The distribution of X The distribution of X with systematic errorwith systematic errorThe distribution of X The distribution of X with systematic errorwith systematic error

Notice that systematic error doesNotice that systematic error doesaffect the average; we call affect the average; we call

this a this a biasbias..

If a Measure Is If a Measure Is Reliable...Reliable...


XX11XX11 XX22XX22

We should see that a person’s score on the same test We should see that a person’s score on the same test given twice is given twice is similarsimilar (assuming the trait being (assuming the trait being

measured isn’t changing).measured isn’t changing).



XX11XX11 XX22XX22

T + eT + e11 T + eT + e22

Recall from true score theory that...Recall from true score theory that...

But, if the scores are similar, But, if the scores are similar, whywhy are they similar? are they similar?



XX11XX11 XX22XX22

T + eT + e11 T + eT + e22

The only thing common to the two measures is the true The only thing common to the two measures is the true score, T. Therefore, the score, T. Therefore, the true score true score must determine the must determine the

reliability.reliability.

Reliability Is...Reliability Is...Reliability Is...Reliability Is...

a a ratioratioa a ratioratio

variance of the true scoresvariance of the true scores

variance of the measurevariance of the measure

var(T)var(T)

var(X)var(X)

Reliability Is...Reliability Is...Reliability Is...Reliability Is...

a a ratioratioa a ratioratio

variance of the true scoresvariance of the true scores

variance of the measurevariance of the measure

We can measure the variance of the observed score, X.We can measure the variance of the observed score, X.The greater the variance, the less reliable the measure.The greater the variance, the less reliable the measure.

This Leads Us to...This Leads Us to...This Leads Us to...This Leads Us to...

We cannot calculate reliability We cannot calculate reliability exactly; we can only exactly; we can only estimateestimate it. it.

Each estimate attempts to capture Each estimate attempts to capture the consequences of the true score the consequences of the true score in different ways.in different ways.

We want both We want both Reliability and Reliability and

ValidityValidity

Reliability and ValidityReliability and Validity

Reliable but not validReliable but not validReliable but not validReliable but not valid


Valid but not reliableValid but not reliableValid but not reliableValid but not reliable


Neither reliable nor validNeither reliable nor validNeither reliable nor validNeither reliable nor valid


Reliable and validReliable and validReliable and validReliable and valid

Assignment #1Assignment #1 Assess the validity and reliability of the Assess the validity and reliability of the

IRIS-3 International Country Risk Guide.IRIS-3 International Country Risk Guide. Can examine a single instance, compare Can examine a single instance, compare

instances, analyze the full variation in the instances, analyze the full variation in the dataset, compare with additional measures, dataset, compare with additional measures, or use any other form of assessment. May or use any other form of assessment. May use outside sources of data, history, or use outside sources of data, history, or analysis (but document).analysis (but document).

The only restriction is that the paper must The only restriction is that the paper must be empirical and examine issues of validity be empirical and examine issues of validity and reliability. and reliability.

3-5 pages. Be concise. 3-5 pages. Be concise. Due Monday 10/24 at beginning of class. Due Monday 10/24 at beginning of class.

the theory of sampling and measurement. sampling first step in implementing any research design is...

Documents

population parameter4

samplingwhat population

inferring population

chance true population

sampling distribution4

theory of sampling

self esteemfrequencythe

asian cases