Research Methodology and Methods of Social Inquiry

www.socialinquiry.wordpress.com

Nov 8, 2011

Assessing Measurement

Reliability & Validity

 

3 sources of variation: 

Observed variation = true differences + systematic measurement error + random measurement error

 

The measure needs to be:

Valid

Reliable

Exhaustive

Mutually Exclusive

VALIDITY

Claims of having appropriately measured the DV and IVs are valid Validity of measurement

Assuming that there is a relationship in this study, if we claim

causality, is the relationship causal? Internal Validity of the causal argument

Generalizations from the results of our study to other units of

observations (e.g. persons) in other places & at other times External Validity of our conclusions

Validity of Measurement

An empirical measure is valid to the extent to which it adequately captures the real meaning of the concept

under consideration

(how well it measures the concept it is intended to measure)

Face validity

Content validity

Criterion-related Validity

Construct Validity

Face Validity

- look at the operationalization; assess whether "on its face" it seems like a good translation of the construct

- improve the quality of face validity assessment: make it

more systematic.

Content Validity

- to what extent does the measure represent all facets of the concept;

- identify clearly the components of the total content ‘domain’; then show that the items adequately represent the components;

Ex: knowledge tests

- assumes a good detailed description of the content domain, which may not always be so.

Criterion-related Validity, I

- applies to measures (‘tests’) that should indicate a person’s present/future standing on a specific behavior (trait)

The behavior (trait) = the criterion;

Validation is a matter of how well scores on the measure correlate with the criterion of interest.

Criterion-related Validity, II

Predictive Validity

- assess how well the measure is able to predict something it should theoretically be able to predict.

Concurrent Validity

- assess how well the measure is able to distinguish between groups that it should theoretically be able to distinguish between.

Construct Validity, I

- based upon accumulation of research evidence (lit. rev)- ‘construct’ <–> ‘concept’

Assumption: the meaning of any concept is implied by statements of its theoretical relation to other concepts

Hence: - Examine theory;

- Hypotheses about variables that should be related to measure(s) of the concept;

- Hypotheses about variables that should NOT be related to measure(s) of the concept;

- Gather ‘evidence’

Construct Validity II

Convergent Validity

- examine the degree to which the measure is similar to (converges on) other measures that it theoretically should be similar to.

Discriminant Validity

- examine the degree to which the measure is not similar to (diverges from) other measures that it theoretically should be not be similar to.

To estimate the degree to which any two measures are related to each other one typically uses the correlation coefficient.

www.socialresearchmethods.net/kb/constval.php - construct validity

RELIABILITY (consistency of measurement)

-deals with the quality of measurement;

A measure = reliable if it would give the same result over & over again

(Assumption: what we are measuring is not changing!)

True Score Theory - every measurement has 2 additive components: true ability (or

the true level) of the respondent on that measure; PLUS random error.

- foundation of reliability theory:

A measure that has no random error (i.e., is all true score) is perfectly reliable; a measure that has no true score (i.e., is all random error) has zero reliability.

Assessing Reliability

Inter-coder reliability:

- check the degree to which different interviewers/observers/raters/coders give consistent estimates of the same phenomenon.

A. Nominal measure raters are checking off which category each observation falls in: calculate % of agreement between raters.

Ex: 1 measure, with 3 categories;

N = 100 observations, rated by 2 raters.

On 86 of the 100 observations, the raters checked the same category (i.e., 86% inter-rater agreement)

B. Measure = continuous: calculate the correlation btw. the

ratings of the 2 raters/observers.

Ex: rating the overall level of activity in a classroom on a 1-

to-7 scale.

Ask raters to give their rating at regular time intervals (e.g., every 60 seconds); the correlation btw. these ratings = estimate of the consistency between the raters.

Test-retest reliability

- administer same test to the same sample on 2 different occasions;

- calculate the correlation btw. repeated applications of the measure through time.

Problems:

- people remember answers;

- real change in attitudes may occur;

- first application of measure may have produced change in the subject.

Internal consistency reliability

-examines the consistency of responses across all items (simultaneously) in a composite measure (uses a single measurement instrument administered to a group of people on 1 occasion to estimate reliability).

How consistent are the results for different items for the same construct within the measure?

Split-half reliability

- calculate the correlation btw. responses to subsets of items from the same measure

(apply scale to sample; then divide scale in 2, randomly; reapply each half; make correlation of results; the higher the better)

- when factors systematically influence the process of measurement, or the concept we measure - systematic measurement error

- when temporary, chance factors affect measurement – random error

- its presence, extent and direction are unpredictable from one question to the next, or from one respondent to the next

www.socialresearchmethods.net

Relation Validity – Reliability – Measurement Error

Systematic error affects distance from center;

Random error affects tightness of pattern AND distance from center

Appropriate Measurement

Valid

Reliabile

Exhaustive- should exhaust the possibilities of what it is intended to

measure;

There must be sufficient categories so that virtually all units of observations being classified will fit into one of the categories.

Mutually exclusive- each observation fits one and only one of the scale values

(categories).