instrumentation. instruments questionnaires surveys interviews how do you know what to ask?
TRANSCRIPT
Instrumentation
Instruments•Questionnaires•Surveys•Interviews
•How do you know what to ask?
measuring variables
•“…a characteristic or attribute of an individual or an organization that can be measured or observed by the researcher and varies among individuals or organizations studied” (Chp 5, p.84)
•Variables are measured using a measurement instrument, a tool, a data collection instrument or just an instrument
Levels of measurement
•Hierarchical•Determine what statistical analysis can be
used
•NOMINAL•ORDINAL•INTERVAL•RATIO
Lowest Level
Highest Level
NOMINAL (categorical)
• Names, nomenclature, or labels used to classify
• Two requirements:▫ Categories must be mutually exclusive▫ Categories have to be exhaustive
• Nominal measures do not convey any value to what is measured …just name it (e.g. race, sex, class)
• Tests of difference (nonparametric):▫ Chi-square
• Measure of association (correlation):▫ Contingency coefficient
Ordinal (categorical)
•Three requirements:▫Categories have to be mutually exclusive▫Categories must be exhaustive▫Categories allow rank ordering
Categories represent relatively more or less of something, however the distance between categories cannot be measured
•How would you describe your level of health?▫Excellent▫Good▫Fair ▫Poor
Ordinal cont.
•Nonparametric statistics
▫Correlational coefficients
Spearman r
Kendal r
Interval (continuous)
• Category Rules:▫Mutually exclusive▫Exhaustive▫Rank order
Widths of categories must be the same which allows for the distance between categories to be measured
No absolute zero (e.g. temperature)
• Parametric statistics:▫Means, standard deviations▫Pearson correlations▫ t-test▫F-test
Ratio (continuous)
•Category Rules:▫Mutually exclusive▫Exhaustive▫Rank order
Widths of categories must be the same which allows for the distance between categories to be measured
▫Scale has an absolute zero (e.g. age, height, weight, time)
•Sometimes referred to a numerical•All statistical tests
Psychometric properties
•Accuracy = validity
•Consistency = reliability
•Fairness = appropriate for participants
Fairness•Has traditionally been given the least
attention•No one way to measure fairness• Is the instrument “fair” for individuals of
various ethnic groups, educational levels, gender, etc.
Examples:•Cultural and language sensitivity
▫ Assessment of conceptual and linguistic equivalence of other language versions of an instrument
•Literacy ▫Evaluate the reading level of the instrument
Validity = accuracy
•Does the instrument accurately measure what it is intending to measure?
•Establishing Validity of an Instrument:1. Criterion-related validity2. Construct validity3. Content validity4. Face validity
Criterion-related validity
• Data from a measurement instrument is correlated with data generated from a measure (criterion) of the concept being studied (usually an individual’s performance or behavior).
• Use of a second measure of a concept as a criterion by which the validity of the new measure can be checked.
• Two types:▫ Predictive validity – measure used will be
correlated with a future performance or behavior LSAT scores accurately predict success in law school
in the future SAT and ACT scores predict future college success
Concurrent validity – a new instrument and an established (valid) instrument that measure the same thing are given to the same sample and the results of the new instrument correlate with the results of the established instrument Beck’s Depression Inventory (established )
and the BDI II (new) High positive correlation between scores is
evidence of concurrent validity
Criterion-related validity
Construct validity
• If there is no existing instrument to compare to or the concept being measured is more abstract in nature, then construct validity is useful.
•The degree to which a measure correlates with other measures it is theoretically expected to correlate with…(driven by theory)
•Are measures positively or negatively associated with each other as would be expected by theory
• Two types:▫ Convergent - degree to which two measures
which purport to be measuring the same topic positively correlate (converge) Theory: Would we expect the relationship to be? Instrument #1 measures person’s self-efficacy for
regular exercise Instrument #2 measures person’s exercise behavior
▫ Discriminant – construct measured (self-efficacy for regular exercise) should not correlate with dissimilar variables (based on theory) Measure of person’s self-efficacy for regular exercise
would not be expected to positively correlate with a person’s inactivity
Construct validity
Content validity
•Content validity is established during an instruments early development, not after completion.
•Assessment of the correspondence between the items that make up an instrument and the content domain from which the items were selected
•Step-by-step process
Face validity
• Weakest form of validity
• Can be a good first step to establishing validity, but it should not replace other means for establishing validity
• “On the face” the instrument appears to measure what it says it measures
• Established by having individuals familiar with the concept look over the instrument to see if it appears to cover the concept it seeks to measure
Validity of screening and diagnostic tests
•Sensitivity – ability of the test to identify correctly all screened individuals who actually have the disease/condition
a true positives
a + c true positives + false negatives
•Specificity – the ability of the test to identify only nondiseased individuals who actually do not have the disease/condition
d true negatives
b + d false positives + true negatives
Validity is most important
If an instrument does not measure what it is supposed to (validity), then it does not matter if it is reliable.
Reliability
•Consistency – extent to which a measure will produce the same or nearly the same results each time it is used
•Reliability is estimated by computing a correlation coefficient (r) ..the closer the correlation coefficient is to 1.00, the greater the reliability▫Values of r between .6 and .8 = moderate▫Values of r above .8 = substantial
correlation
Methods for determining reliability
•Parallel Forms•Internal Consistency•Test-Retest•Rater
▫Interrator▫intrarater
Parallel Forms•Also called equivalent form or
alternate form
•Create different forms of the same measure that when given to the same participants will produce similar results (means, SD, item correlations)
•Example: SAT and ACT exams
Internal consistency
• One of the most common methods used
• Intercorrelations among individual items
• Correlate each individual item and the total score
• The greater the correlation, the higher the reliability
• Statistical Measures:▫Cronbach’s alpha▫Kuder-Richardson(KR) 20 or 21 coefficient▫Spearman-Bowman split-half reliability
Test-retest•Also called stability reliability as it provides
evidence of stability over time
•Same instrument, same group, same conditions, at two different points in time
•Data from two administrations of the instrument are used to calculate a correlation coefficient
•How much time should there be between administations?
Inter- and intra- rater reliability
•Inter = Consistency of an observed event by different raters (e.g. three research team members are identifying themes from interview transcripts as part of a qualitative study)
•Intra = consistent measurement or rating by the same person (e.g. I take blood pressure measures of community members)