instrumentation. instruments questionnaires surveys interviews how do you know what to ask?

Instrumentation

Instruments•Questionnaires•Surveys•Interviews

•How do you know what to ask?

measuring variables

•“…a characteristic or attribute of an individual or an organization that can be measured or observed by the researcher and varies among individuals or organizations studied” (Chp 5, p.84)

•Variables are measured using a measurement instrument, a tool, a data collection instrument or just an instrument

Levels of measurement

•Hierarchical•Determine what statistical analysis can be

used

•NOMINAL•ORDINAL•INTERVAL•RATIO

Lowest Level

Highest Level

NOMINAL (categorical)

• Names, nomenclature, or labels used to classify

• Two requirements:▫ Categories must be mutually exclusive▫ Categories have to be exhaustive

• Nominal measures do not convey any value to what is measured …just name it (e.g. race, sex, class)

• Tests of difference (nonparametric):▫ Chi-square

• Measure of association (correlation):▫ Contingency coefficient

Ordinal (categorical)

•Three requirements:▫Categories have to be mutually exclusive▫Categories must be exhaustive▫Categories allow rank ordering

Categories represent relatively more or less of something, however the distance between categories cannot be measured

•How would you describe your level of health?▫Excellent▫Good▫Fair ▫Poor

Ordinal cont.

•Nonparametric statistics

▫Correlational coefficients

Spearman r

Kendal r

Interval (continuous)

• Category Rules:▫Mutually exclusive▫Exhaustive▫Rank order

Widths of categories must be the same which allows for the distance between categories to be measured

No absolute zero (e.g. temperature)

• Parametric statistics:▫Means, standard deviations▫Pearson correlations▫ t-test▫F-test

Ratio (continuous)

•Category Rules:▫Mutually exclusive▫Exhaustive▫Rank order

Widths of categories must be the same which allows for the distance between categories to be measured

▫Scale has an absolute zero (e.g. age, height, weight, time)

•Sometimes referred to a numerical•All statistical tests

Psychometric properties

•Accuracy = validity

•Consistency = reliability

•Fairness = appropriate for participants

Fairness•Has traditionally been given the least

attention•No one way to measure fairness• Is the instrument “fair” for individuals of

various ethnic groups, educational levels, gender, etc.

Examples:•Cultural and language sensitivity

▫ Assessment of conceptual and linguistic equivalence of other language versions of an instrument

•Literacy ▫Evaluate the reading level of the instrument

Validity = accuracy

•Does the instrument accurately measure what it is intending to measure?

•Establishing Validity of an Instrument:1. Criterion-related validity2. Construct validity3. Content validity4. Face validity

Criterion-related validity

• Data from a measurement instrument is correlated with data generated from a measure (criterion) of the concept being studied (usually an individual’s performance or behavior).

• Use of a second measure of a concept as a criterion by which the validity of the new measure can be checked.

• Two types:▫ Predictive validity – measure used will be

correlated with a future performance or behavior LSAT scores accurately predict success in law school

in the future SAT and ACT scores predict future college success

Concurrent validity – a new instrument and an established (valid) instrument that measure the same thing are given to the same sample and the results of the new instrument correlate with the results of the established instrument Beck’s Depression Inventory (established )

and the BDI II (new) High positive correlation between scores is

evidence of concurrent validity

Criterion-related validity

Construct validity

• If there is no existing instrument to compare to or the concept being measured is more abstract in nature, then construct validity is useful.

•The degree to which a measure correlates with other measures it is theoretically expected to correlate with…(driven by theory)

•Are measures positively or negatively associated with each other as would be expected by theory

• Two types:▫ Convergent - degree to which two measures

which purport to be measuring the same topic positively correlate (converge) Theory: Would we expect the relationship to be? Instrument #1 measures person’s self-efficacy for

regular exercise Instrument #2 measures person’s exercise behavior

▫ Discriminant – construct measured (self-efficacy for regular exercise) should not correlate with dissimilar variables (based on theory) Measure of person’s self-efficacy for regular exercise

would not be expected to positively correlate with a person’s inactivity

Construct validity

Content validity

•Content validity is established during an instruments early development, not after completion.

•Assessment of the correspondence between the items that make up an instrument and the content domain from which the items were selected

•Step-by-step process

Face validity

• Weakest form of validity

• Can be a good first step to establishing validity, but it should not replace other means for establishing validity

• “On the face” the instrument appears to measure what it says it measures

• Established by having individuals familiar with the concept look over the instrument to see if it appears to cover the concept it seeks to measure

Validity of screening and diagnostic tests

•Sensitivity – ability of the test to identify correctly all screened individuals who actually have the disease/condition

a true positives

a + c true positives + false negatives

•Specificity – the ability of the test to identify only nondiseased individuals who actually do not have the disease/condition

d true negatives

b + d false positives + true negatives

Validity is most important

If an instrument does not measure what it is supposed to (validity), then it does not matter if it is reliable.

Reliability

•Consistency – extent to which a measure will produce the same or nearly the same results each time it is used

•Reliability is estimated by computing a correlation coefficient (r) ..the closer the correlation coefficient is to 1.00, the greater the reliability▫Values of r between .6 and .8 = moderate▫Values of r above .8 = substantial

correlation

Methods for determining reliability

•Parallel Forms•Internal Consistency•Test-Retest•Rater

▫Interrator▫intrarater

Parallel Forms•Also called equivalent form or

alternate form

•Create different forms of the same measure that when given to the same participants will produce similar results (means, SD, item correlations)

•Example: SAT and ACT exams

Internal consistency

• One of the most common methods used

• Intercorrelations among individual items

• Correlate each individual item and the total score

• The greater the correlation, the higher the reliability

• Statistical Measures:▫Cronbach’s alpha▫Kuder-Richardson(KR) 20 or 21 coefficient▫Spearman-Bowman split-half reliability

Test-retest•Also called stability reliability as it provides

evidence of stability over time

•Same instrument, same group, same conditions, at two different points in time

•Data from two administrations of the instrument are used to calculate a correlation coefficient

•How much time should there be between administations?

Inter- and intra- rater reliability

•Inter = Consistency of an observed event by different raters (e.g. three research team members are identifying themes from interview transcripts as part of a qualitative study)

•Intra = consistent measurement or rating by the same person (e.g. I take blood pressure measures of community members)