Designing a Study:Designing a Study:Critical Evaluation and Critical Evaluation and

MeasurementMeasurement

OverviewOverview ReliabilityReliability ValidityValidity Demand CharacteristicsDemand Characteristics Measurement ScalesMeasurement Scales

ReliabilityReliability

Reliability: Reliability: the degree to which the the degree to which the same event or measure produces the same event or measure produces the same result each time it is measured.same result each time it is measured.

Any Measure = True Score + Random ErrorAny Measure = True Score + Random Error

Reliability of Measures

Less error

Reliability of MeasuresReliability of Measures Methods of assessing reliability

1. Test-retest reliability

2. Internal consistency reliability• Split-half reliability

• Cronbach’s alpha

3. Interrater reliability

Remember…Remember…

A A reliablereliable test is not test is not necessarily necessarily valid!valid!

Threats to Internal ValidityThreats to Internal Validity

Selection/Volunteer biasSelection/Volunteer bias

Regression towards the meanRegression towards the mean

Repeated testingRepeated testing

Attrition/MortalityAttrition/Mortality

Construct ValidityConstruct Validity

Construct validity- Is the measure that is used actually measuring the construct it is intended to measure?

Indicators of Construct ValidityIndicators of Construct Validity Face Validity: it looks good.Face Validity: it looks good.

Concurrent Validity: it matches current Concurrent Validity: it matches current behaviorbehavior

Predictive Validity: it predicts future behaviorPredictive Validity: it predicts future behavior

Convergent Validity: it matches other Convergent Validity: it matches other measuresmeasures

Discriminant Validity: it doesn’t fit other Discriminant Validity: it doesn’t fit other measuresmeasures

Demand CharacteristicsDemand Characteristics

extraneous cues that guide a participant’s behavior

reactivity:: A measure is reactive if awareness of being measured changes an individual’s behavior

social desirability: participants may behave in ways they think are more ‘acceptable’

How can we How can we reducereduce demand demand characteristics?characteristics?

unobtrusive measuresunobtrusive measures e.g. one-way mirrors, hidden cameras, e.g. one-way mirrors, hidden cameras,

prisms in periscopes under the bathroom prisms in periscopes under the bathroom stall door…stall door…

deceptiondeception a cover story to disguise the true purposea cover story to disguise the true purpose

distractor or filler tasksdistractor or filler tasks keeps attention away from purposekeeps attention away from purpose

Variables and Measurement ScalesVariables and Measurement Scales

Nominal/Categorical Scales

Ordinal Scales

Interval Scales

Ratio Scales

Identify the measurement scale for each of the following data:

1. Circle your marital status: Married Single Divorced Engaged

2. Do you go to work today? Yes No

3. If you work, how many hours a week do you work? _______

4. Rate the extent that you enjoy college on the scale below.

1 2 3 4 5

Not at all Very much

5. Name your top 3 favorite TV shows in order of preference:

1. ___________

2. ___________

3. ____________