7 measurement & questionnaires design (dr. mai,2014)
TRANSCRIPT
7.Measurement & Questionnaires Design
Dr. Nguyen Quynh Mai
Measurement
Give examples:
• What are some things that are easy to measure?
• What are some things that are difficult to measure?
Measurement
• Quantitative measurement involves assigning numbers to
attributes or constructs in order to create variables so that
we can better understand the relationships among
variables
• With qualitative inquiry involves assigning words to attributes
or concepts in order to draw out themes related to human
perceptions and meaning
Measurement
• If our studies do not allow us to measure variation in the
dependent variable (Y) as related to variation in our X
variables, then we cannot do any scientific testing.
1. We measure whether certain variables are meaningful –
individually significant.
2. We measure the variation in our variables.
3. We also measure the significance and explanatory power of our
models and the relationships between variables.
4. If it can be quantified, then you should do so.
Qualities of Variables
• Exhaustive -- Should include all possible answerable responses. (Schooling: No Schooling, Elementary, Middle, HS, College)
• Mutually exclusive -- No respondent should be able to have two attributes simultaneously (e.g. Female Male ).
What Is Level of Measurement?
The relationship of the values that are assigned to the
attributes for a variable
1 2 3
Relationship
Values
Attributes
Variable
Low Medium High
Development
8
Types of Scales
Ordinal
interval
Ratio
Nominal
9
Levels of Measurement
Ordinal
interval
Ratio
NominalClassification
Order
Classification
Order
Classification Distance
Natural Origin
Order
Classification Distance
10
Nominal Scales• Nominal or categorical data is data that comprises of categories
that cannot be rank ordered – each category is just different
• Mutually exclusive and collectively exhaustive categories
• Exhibits the classification characteristic only. Nominal data reflect
qualitative differences rather than quantitative ones.
• Variables that have only two responses i.e. Yes or No, are known as
dichotomies
• Examples:What is your gender? (please tick)
Male
Female
Did you enjoy the film? (please tick)
Yes
No
11
Ordinal Scales• Characteristics of nominal scale plus
an indication of order
• Implies statement of greater than
and less than
• Ex: Doneness of meat (well, medium
well, medium rare, rare)
• Both interval and ratio data are examples of scale data.
• Scale data:
data is in numeric format (£50, £100, £150)
data that can be measured on a continuous scale
the distance between each can be observed and as a result
measured
the data can be placed in rank order.
Interval and ratio data
13
Interval Scales
• Interval data measured on a continuous scale and has
no true zero point
• Characteristics of nominal and ordinal scales plus the
concept of equality of interval.
• Equal distance exists between numbers
• Examples:
• Time – moves along a continuous measure or seconds, minutes and so on and is without a zero point of time.
• Temperature – moves along a continuous measure of degrees and is without a true zero.
14
Ratio Scales
• Characteristics of previous scales plus an absolute zeropoint
• Examples
– Weight
– Height
– Number of children
– Age
Measurement Hierarchy
NOMINAL
ORDINAL
INTERVAL
RATIO
WEAKEST
STRONGEST
Sources of Error
Respondent
InstrumentMeasurer
Situation
Evaluating Measurement Tools
Criteria
Validity
Practicality Reliability
18
Reliability & Validity
19
Validity Determinants
Content
ConstructCriterion
20
Increasing Content Validity
ContentLiterature
Search
Expert
Interviews
Group
Interviews
Question
Database
Etc.
21
Increasing Construct Validity
New measure of trust
Known measure of trust
Empathy
Credibility
22
Judging Criterion Validity
Relevance
Freedom from bias
Reliability
Availability
Criterion
23
Reliability Estimates
Stability
Internal
ConsistencyEquivalence
Reliability• Reliability is defined as the extent to which a questionnaire, test, observation
or any measurement procedure produces the same results on repeated trials
• Equivalence refers to the amount of agreement between two or more instruments that are administered at nearly the same point in time. Equivalence is measured through a parallel forms procedure in which one administers alternative forms of the same measure to either the same group or different group of respondents.
• Stability occurs when the same or similar scores are obtained with repeated testing with the same group of respondents. Stability is assessed through a test-retest procedure that involves administering the same measurement instrument to the same individuals under the same conditions after some period of time.
• Internal consistency concerns the extent to which items on the test or instrument are measuring the same thing. If the individual items are highly correlated with each other you can be highly confident in the reliability of the entire scale
(Miler, Western International University) 24
Equivalence test• Reliability is established by using similar/alternate forms (Forms A
& B) that measure the same trait/knowledge.
• Two forms are created by splitting the questions on the test
randomly before administration of the forms. One group of
students gets form "A" first, then "B". Another group takes the "B"
form of the test first, then the "A" version. The scores on both are
then correlated, producing a correlation or reliability coefficient.
25
Stability test: Test-Retest
• If you use a test with a student in the morning and then administer it again in the afternoon, would you expect about the same results?
• Which would have the highest test-retest reliability coefficient (correlation)?
– retesting a youngster on an achievement test within the same week
– retesting in different years
26
Internal consistency• The internal consistency reliability of survey instruments is a measure of
reliability of different survey items intended to measure the same
characteristic.
• For example, there are 5 different questions (items) related to anxiety
level. Each question implies a response with 5 possible values on a Likert
scale (1 – 5). Responses from a group of respondents have been
obtained. In reality, answers to different questions vary for each
particular respondent, although the items are intended to measure the
same aspect or quantity. The stronger the correlation, the greater the
internal consistency reliability of this survey instrument.
• In statistic, they use Cronbach alpha to measure reliability
27
28
Practicality
Economy InterpretabilityConvenience
MEASUREMENT SCALES
30
Nature of Attitudes: example of measurement
CognitiveI think oatmeal is healthier
than corn flakes for breakfast.
Affective/
attitude
Behavioral
I hate corn flakes.
I intend to eat more oatmeal
for breakfast.
31
Improving Predictability
Reference
groups
Multiple
measures
Factors
Strong
Specific
Basis
Direct
32
Selecting a Measurement Scale
Research objectives Response types
Data propertiesNumber of
dimensions
Forced or unforced
choices
Balanced or
unbalanced
Rater errorsNumber of
scale points
33
Response Types
Rating scale
Ranking scale
Categorization
Sorting
Number of Dimensions
Unidimensional
Multi-dimensional
Balanced or Unbalanced
Very bad
Bad
Neither good nor bad
Good
Very good
Poor
Fair
Good
Very good
Excellent
How good an actress is Angelina Jolie?
Forced or Unforced Choices
Very bad
Bad
Neither good nor bad
Good
Very good
Very bad
Bad
Neither good nor bad
Good
Very good
No opinion
Don’t know
How good an actress is Angelina Jolie?
Number of Scale Points
Very bad
Bad
Neither good nor bad
Good
Very good
Very bad
Somewhat bad
A little bad
Neither good nor bad
A little good
Somewhat good
Very good
How good an actress is Angelina Jolie?
Simple Category Scale
I plan to purchase a Samsung laptop in the
12 months.
Yes
No
Multiple-Choice, Single-Response Scale
What newspaper do you read
most often for financial news?
East City Gazette
West City Tribune
Regional newspaper
National newspaper
Other (specify:_____________)
Multiple-Choice, Multiple-Response Scale
What sources did you use when designing your new
home? Please check all that apply.
Online planning services
Magazines
Independent contractor/builder
Designer
Architect
Other (specify:_____________)
Likert Scale
The Internet is superior to traditional libraries for
comprehensive searches.
Strongly disagree
Disagree
Neither agree nor disagree
Agree
Strongly agree
Semantic Differential
SD Scale for Analyzing Actor Candidates
Numerical Scale
Multiple Rating List Scales
Ranking Scales
• Paired-comparison scale
• Forced ranking scale
• Comparative scale
Paired-Comparison Scale
Forced Ranking Scale
Questionnaires Design
Dr. Nguyen Quynh Mai
Overall Flowchart for Instrument Design
Flowchart for Instrument Design: Phase 1
Strategic Concerns in Instrument Design
What type of scale is needed?
What communication approach will be used?
Should the questions be structured?
Should the questioning be disguised?
Flowchart for Instrument Design: Phase 2
Question Categories and Structure
Administrative
Classification
Target
Question Content
Should this question be asked?
Is the question of proper scope and coverage?
Can the participant adequately
answer this question as asked?
Will the participant willingly
answer this question as asked?
Question Wording
Criteria
Shared
vocabularySingle
meaning
Misleading
assumptions
Adequate
alternatives
Personalized
Biased
Response Strategy
Factors
Objectives
of the study
Participant’s
level of
information
Degree to which
participants have
thought through topic
Ease and clarity with
which participant
communicates
Participant’s
motivation to
share
Free-Response Strategy
What factors influenced your enrollment in Metro U?
____________________________________________
____________________________________________
Dichotomous Response Strategy
Did you attend the “A Day at College”
program at IU?
Yes
No
Multiple Choice Response Strategy
Which one of the following factors was most influential
in your decision to attend Metro U?
Good academic standing
Specific program of study desired
Enjoyable campus life
Many friends from home
High quality of faculty
Checklist Response Strategy
Which of the following factors influenced your
decision to enroll in Metro U? (Check all that
apply.)
Tuition cost
Specific program of study desired
Parents’ preferences
Opinion of brother or sister
Many friends from home attend
High quality of faculty
Rating Response Strategy
Strongly
influential
Somewhat
influential
Not at all
influential
Good academic reputation
Enjoyable campus life
Many friends
High quality faculty
Semester calendar
RankingPlease rank-order your top three factors from the following
list based on their influence in encouraging you to apply to
Metro U. Use 1 to indicate the most encouraging factor, 2
the next most encouraging factor, etc.
_____ Opportunity to play collegiate sports
_____ Closeness to home
_____ Enjoyable campus life
_____ Good academic reputation
_____ High quality of faculty
Summary of Scale TypesType Restrictions Scale
Items
Scale
points
Data Type
Rating Scales
Simple Category
Scale
Needs mutually exclusive choices One or
more
2 Nominal
Multiple Choice
Single-Response
Scale
Needs mutually exclusive choices;
may use exhaustive list or ‘other’
many 2 Nominal
Multiple Choice
Multiple-Response
Scale (checklist)
Needs mutually exclusive choices;
needs exhaustive list or ‘other’
many 2 Nominal
Likert Scale Needs definitive positive or
negative statements with which to
agree/disagree
One or
more
5 Ordinal
Likert-type Scale Needs definitive positive or
negative statements with which to
agree/disagree
One or
more
7 or 9 Ordinal
Summary of Scale Types
Type Restrictions Scale
Items
Scale
points
Data Type
Semantic
Differential
Scale
Needs words that are opposites to
anchor the graphic space.
One or
more
7 Ordinal
Numerical
Scale
Needs concepts with standardized or
defined meanings; needs numbers
anchor the end-points or points along
the scale; score is a measurement of
graphical space from one anchor.
One or
many
3-10 Ordinal or
Interval
Multiple Rating
List Scale
Needs words that are opposites to
anchor the end-points on the verbal
scale
Up to 10 5-7 Ordinal
Fixed Sum
Scale
Participant needs ability to calculate
total to some fixed number, often 100.
Two or
more
none Interval or
Ratio
Stapel Scale Needs verbal labels that are
operationally defined or standard.
One or
more
10 Ordinal or
Interval
Summary of Scale Types
Type Restrictions Scale
Items
Scale
points
Data Type
Ranking Scales
Graphic Rating
Scale
Needs visual images that can be
interpreted as positive or negative
anchors; score is a measurement of
graphical space from one anchor.
One or
more
none Ordinal
(Interval, or
Ratio)
Paired
Comparison
Scale
Number is controlled by participant’s
stamina and interest.
Up to 10 2 Ordinal
Forced
Ranking Scale
Needs mutually exclusive choices. Up to 10 many Ordinal or
Interval
Comparative
Scale
Can use verbal or graphical scale. Up to 10 Ordinal
Internet Survey Scale Options
Internet Survey Scale Options
Internet Survey Scale Options
Flowchart for Instrument Design: Phase 3
Guidelines for Question Sequencing
Interesting topics early
Classification questions later
Sensitive questions later
Simple items early
Transition between topics
Reference changes limited
Components of Questionnaires
Introduction
Transition
Instructions for ….
Conclusion
a. Terminating
b. Participant discontinue
c. Skip
d. Disposition instruction
Tips for a good questionnaires
1. Keep it short and simple
2. Start with a brief introduction explaining the purpose of the
questionnaire
3. Ask yourself what you will do with the information from each
question
4. Put easier questions first
5. Ask just one thing in a question
6. Leave difficult or sensitive questions towards the end
7. Try to be exhaustive when offering answer choices
8. Avoid bias in language
Tips for a good questionnaires (Cont)
9. Avoid jargon (special words that are used by a particular
profession group and are difficult for others to understand)
10. Do not use "emotional language" or leading questions
11. Present Disagree/Agree choices in that order (disagree to agree);
Same with Excellent to Poor, and Positive to Negative
12. User higher numbers to signify a more positive answer in rating
scales
13. Be consistent with your layout
14. Separate related questions
15. Have Don’t Know or Not Applicable as an option for most
questions
Tips for a good questionnaires (Cont)
16. Have Other or None for questions with a list of options
17. Include Other Comments at the end
18. Keep the number of open-ended questions to a minimum
19. Make sure questions are relevance with scale
20. Pilot test the questionnaire