“the proof is in the pudding, the perils are in the plan”data collection methods “the proof is...
TRANSCRIPT
![Page 1: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/1.jpg)
Data Collection Methods
“The Proof is in the Pudding, the Perils are in the Plan”
Melissa Chapman Haynes, Ph.D.
Senior Evaluator, Professional Data Analysts, Inc.
MESI Spring Training 2017
![Page 2: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/2.jpg)
That look when you are trying to analyze data collected without thoughtful planning…
![Page 3: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/3.jpg)
3Prepared by Professional Data Analysts, Inc.
LEARNING OBJECTIVESParticipants will…
Gain a deeper understanding of how to develop
and align evaluation questions, approaches, and
methods that are feasible and appropriate to the
context.
Recognize the considerations, advantages, and
limitations of various types of data collection
methods
Identify technologies available to support data
collection
Gain a tool for communicating data collection
plans with team members and/or clients + increase
buy-in from key stakeholders
Minnesota Evaluation Studies Institute Spring Training 2017
![Page 4: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/4.jpg)
![Page 5: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/5.jpg)
5Prepared by Professional Data Analysts, Inc.
SCOPE – WHAT ARE WE DOING TODAY?
Types of Data Collection
5
3
4
2
Sampling
Innovations and Technologies
Pulling Together and Sharing the Plan
Context, Questions, and Feasibility1
![Page 6: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/6.jpg)
Section 1: Context, Questions, & Feasibility
![Page 7: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/7.jpg)
7Prepared by Professional Data Analysts, Inc.
DATA COLLECTION IN THE EVALUATION CYCLE
“Compiling information that stakeholders perceive
as trustworthy and relevant for answering their
questions. Such evidence can be experimental or
observational, qualitative or quantitative, or it can
include a mix of methods. Adequate data might be
available and easily accessed, or it might need to
be defined and new data collected. Whether a body
of evidence is credible to stakeholders might
depend on such factors as how the questions were
posed, sources of information, conditions of data
collection, reliability of measurement, validity of
interpretations, and quality control procedures.”
Minnesota Evaluation Studies Institute Spring Training 2017
![Page 8: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/8.jpg)
8Prepared by Professional Data Analysts, Inc.
WHAT COUNTS AS CREDIBLE EVIDENCE?
Demands for such evidence is increasing
Program-theory driven
- Develop program impact theory
- Develop evaluation questions
- Answer evaluation questions
What constitutes knowledge and how it is
created? (Patton, 2002)
- Positivism (quantitative)
- Constructivism (qualitative)
Minnesota Evaluation Studies Institute Spring Training 2017
![Page 9: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/9.jpg)
9Prepared by Professional Data Analysts, Inc.
ALL DATA COLLECTION IS A TYPE OF CONVERSATION, A RELATIONSHIP
With each question
asked, reflect: How will
this information be
used? Is it necessary for
us to ask?
Know your potential
respondents – know the
context (language,
reading levels, preferred
communication formats,
etc.)
![Page 10: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/10.jpg)
10Prepared by Professional Data Analysts, Inc.
START WITH THE EVALUATION QUESTIONS, NOT THE METHODS
Minnesota Evaluation Studies Institute Spring Training 2017
How many times have you
heard someone say, “Let’s
do a survey – it’s easy!”
Process Evaluation
To whom did you direct program efforts?
What has your program done?
When did your program activities take place?
What are the barriers/facilitators to implementation of program
activities?
Outcome Evaluation
Were the providers who received training more likely to
effectively screen and treat patients than those who did not?
Did the program have any unintended effects on the target
population?
Do the benefits of the program justify a continued allocation of
resources?
![Page 11: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/11.jpg)
11Prepared by Professional Data Analysts, Inc.
EVALUATING PROCESS
![Page 12: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/12.jpg)
12Prepared by Professional Data Analysts, Inc.
EVALUATING RESULTS
![Page 13: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/13.jpg)
13Prepared by Professional Data Analysts, Inc.
WHAT IS YOUR INQUIRY WORLDVIEW?
What was your experience in completing the questionnaire? Was it
difficult? Why?
Did your score surprise you? Why or why not?
How might one’s worldview affect his or her evaluation practice?
How might it have affected your practice with previous evaluations?
How would you deal with a client, colleague, or team member who
had a worldview different from yours?
Minnesota Evaluation Studies Institute Spring Training 2017
![Page 14: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/14.jpg)
14Prepared by Professional Data Analysts, Inc.
FOUR CONSIDERATIONS FOR QUALITY
Credibility of Findings
(who are the
stakeholders and what
do they value?)
• Community members
• Legislators
• Program staff
• Program participants
To be discussed during early planning of the evaluation and data collection processes!
Staff skills
• Project management
• Interpersonal skills –
buy-in from
stakeholder groups
• Data cleaning
• Data management
• Data analysis
• Reporting and
visualization
Costs
• Software
• Training
• Consultants or
collaborations
• Weighing accuracy
with feasibility
• Access to information
(e.g., secondary data)
Time constraints
• Balancing deadlines
with ability to plan and
implement quality
processes
• Communication &
shared understanding
of data collection
process
• IRB considerations &
timelines
Minnesota Evaluation Studies Institute Spring Training 2017
![Page 15: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/15.jpg)
15Prepared by Professional Data Analysts, Inc.
BAD DATA IS NOT BETTER THAN NO DATA
![Page 16: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/16.jpg)
Section 2: Types of Data Collection
![Page 17: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/17.jpg)
17Prepared by Professional Data Analysts, Inc.
TYPES OF DATA COLLECTION
Minnesota Evaluation Studies Institute Spring Training 2017
Surveys Interviews &
focus groups
Observation
![Page 18: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/18.jpg)
18Prepared by Professional Data Analysts, Inc.
TYPES OF DATA COLLECTION
Minnesota Evaluation Studies Institute Spring Training 2017
Extraction Secondary
data
Innovative
strategies
![Page 19: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/19.jpg)
19Prepared by Professional Data Analysts, Inc.
WHEN SHOULD I USE A SURVEY?
![Page 20: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/20.jpg)
20Prepared by Professional Data Analysts, Inc.
Gather self-reported information on knowledge, behaviors, beliefs, experiences
Population is defined
Limited or well-defined need for detailed information
Anonymity possible
PURPOSE: WHEN IS A SURVEY THE APPROPRIATE METHOD?
![Page 21: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/21.jpg)
21Prepared by Professional Data Analysts, Inc.
ITEM TYPES
![Page 22: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/22.jpg)
22Prepared by Professional Data Analysts, Inc.
Likert-type (rating scale)
Dichotomous (T/F or Y/N)
Multiple choice, ranking, rating scales
CLOSED-ENDED
![Page 23: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/23.jpg)
23Prepared by Professional Data Analysts, Inc.
![Page 24: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/24.jpg)
24Prepared by Professional Data Analysts, Inc.
![Page 25: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/25.jpg)
25Prepared by Professional Data Analysts, Inc.
![Page 26: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/26.jpg)
26Prepared by Professional Data Analysts, Inc.
![Page 27: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/27.jpg)
27Prepared by Professional Data Analysts, Inc.
Use: When uncertain of entire range of
alternative answers or when you
want to gather examples, stories,
lists, or descriptions
OPEN-ENDED ITEMS
![Page 28: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/28.jpg)
28Prepared by Professional Data Analysts, Inc.
![Page 29: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/29.jpg)
29Prepared by Professional Data Analysts, Inc.
![Page 30: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/30.jpg)
30Prepared by Professional Data Analysts, Inc.
SURVEY MODE:
PAPER
PHONE
ONLINE
![Page 31: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/31.jpg)
31Prepared by Professional Data Analysts, Inc.
IMPERSONALITY
COGNITIVE BURDEN
LEGITIMACY
Data
Quality
![Page 32: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/32.jpg)
32Prepared by Professional Data Analysts, Inc.
SEVEN DESIGN PRINCIPLES TO REDUCE COGNITIVE BURDEN
Take advantage of knowledge that is available externally + long-term memory
Simplify the structure of tasks
Make both controls and actions that perform visible to the user
Rely on natural mappings between actions and their consequences
Exploit physical and cultural constraintsAllow for errors
Rely on standardization when other design principles do not apply
![Page 33: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/33.jpg)
33Prepared by Professional Data Analysts, Inc.
![Page 34: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/34.jpg)
34Prepared by Professional Data Analysts, Inc.
PAPER SURVEYS
Self-Administration
Impersonality
Accuracy
Level of Reporting
Rate of Missing Data
Cognitive Burden
Reliability
![Page 35: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/35.jpg)
35Prepared by Professional Data Analysts, Inc.
ONLINE SURVEYS
Online
Legitimacy
Accuracy
Level of Reporting
Cognitive Burden
Rate of Missing Data
Reliability
![Page 36: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/36.jpg)
36Prepared by Professional Data Analysts, Inc.
ADAPTIVE DESIGN
![Page 37: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/37.jpg)
37Prepared by Professional Data Analysts, Inc.
FIVE PRINCIPLES FOR DESIGN OF PAPER OR ONLINE SURVEYS
1. Consistently use graphical elements (e.g., contrast and spacing) to define a clear path through the questionnaire.
2. Use prominent visual guides to redirect respondents when conventions within a questionnaire must change.
3. Place directions where they are easily seen and close to where they are needed.
4. Keep separate pieces of information physically close when they must be connected to be understood.
5. Ask only one question at a time.
![Page 38: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/38.jpg)
38Prepared by Professional Data Analysts, Inc.
CONSISTENCY
Section I Strongly Agree
Agree Disagree
Strongly Disagree
No Response
I am satisfied with X.
I found the content to be useful.
Section II Strongly Disagree
Disagree
Agree Strongly Agree
No Response
I would attend X event again.
I would recommend X to a friend.
![Page 39: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/39.jpg)
39Prepared by Professional Data Analysts, Inc.
AESTHETIC
![Page 40: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/40.jpg)
40Prepared by Professional Data Analysts, Inc.
TELEPHONE SURVEYS
Telephone
Legitimacy
Accuracy
Level of Reporting
Cognitive Burden
Rate of Missing Data
Reliability
![Page 41: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/41.jpg)
41Prepared by Professional Data Analysts, Inc.
WHAT IS AN “OPTIMAL” RESPONSE RATESTRATEGIES FOR INCREASING RESPONSE RATES
Strategies for increasing response rates
![Page 42: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/42.jpg)
42Prepared by Professional Data Analysts, Inc.
Estimating non-response bias
Use – how will the results be used? What
are the stakes?
WHAT IS A “GOOD ENOUGH” RESPONSE RATE?
![Page 43: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/43.jpg)
43Prepared by Professional Data Analysts, Inc.
PRE-INCENTIVES
![Page 44: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/44.jpg)
44Prepared by Professional Data Analysts, Inc.
![Page 45: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/45.jpg)
45Prepared by Professional Data Analysts, Inc.
GRADUATE STUDENT SURVEY
![Page 46: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/46.jpg)
46Prepared by Professional Data Analysts, Inc.
![Page 47: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/47.jpg)
47Prepared by Professional Data Analysts, Inc.
MAKE A LIST OF POSSIBLE INCENTIVES THAT COULD BE USED TO OBTAIN A
HIGH SURVEY RESPONSE RATE
![Page 48: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/48.jpg)
48Prepared by Professional Data Analysts, Inc.
Incentive RespondentPopulation
Positive Aspects orAdvantages
Negative Aspects of Disadvantages
![Page 49: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/49.jpg)
49Prepared by Professional Data Analysts, Inc.
• HOW DID YOU DECIDE WHAT WOULD BE AN EFFECTIVE SURVEY INCENTIVE? WHAT WERE YOUR CRITERIA?
• What issues arose in your discussions concerning
the positive and negative aspects of various
incentives?
• What ethical issues did your group discuss relative
to the use of incentives?
• Under what circumstances might the use of
incentives be inappropriate?
![Page 50: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/50.jpg)
50Prepared by Professional Data Analysts, Inc.
Survey fatigue is real, people…
![Page 51: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/51.jpg)
51Prepared by Professional Data Analysts, Inc.
WHEN MIGHT YOU CONDUCT IN-DEPTH INTERVIEWS?
![Page 52: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/52.jpg)
52Prepared by Professional Data Analysts, Inc.
Interview format:
Unstructured
Semi-structured
Structured
Mode of interviews:
Telephone
In-person
Intercepts
Types of Questions:
Descriptive
Structural
Contrast
Interviews provide more nuanced information, personal stories, ability to probe/follow-up
With each question asked, know the type of information you anticipate receiving
![Page 53: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/53.jpg)
53Prepared by Professional Data Analysts, Inc.
Grand tour questions“I am interested in your life when you were growing up. What was your family like? Where did you live? Where did you go to school?”
Mini-tour questions Specific element/follow-up from grand tour questions: “You’ve told me a lot about your life growing up but you haven’t said much about your parents. Please tell me about your parents. For example, what did they do for a living?”
Example questions“Please give me an example of something he did to make you feel that way.”
Experience questions“Please tell me something you did while you were deployed.”
DESCRIPTIVE QUESTIONS RESULT IN A NARRATIVE
![Page 54: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/54.jpg)
54Prepared by Professional Data Analysts, Inc.
Cover term questions“What are the different software options that are authorized to be on computers at this location?”
Included term/item questions “I’ve hear you say that you are allowed to have word processing software and
spreadsheet software on your computer. What are the database applications that are also
approved? What about qualitative coding software?”
Card or list sorting questions“Here are some cards labeled with software names. Can you divide them by type
(spreadsheet, database, word processing)? Which of these are authorized to be on your
machines? Can you divide them by how frequently you use them? “
STRUCTURAL QUESTIONS RESULT IN A LIST
![Page 55: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/55.jpg)
55Prepared by Professional Data Analysts, Inc.
Contrast verifications“So you are permitted to have Microsoft Word but not NVivo?”
Directed contrasts“So you have Word, Excel, and Power Point on your machine. What other
software do you have?”
Rating questionsEstablish the order or value of items, themes, activities, etc.: “What was the
best birthday gift you ever received? What was the worst?”
CONTRAST QUESTIONS MANIPULATE A LIST
![Page 56: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/56.jpg)
56Prepared by Professional Data Analysts, Inc.
WATCH OUT FOR:
Double barreled items
Leading items
Double negatives
Jargon or technical terms
Vague items
Items using emotional language
Items beyond the respondent’s capability to answer
![Page 57: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/57.jpg)
57Prepared by Professional Data Analysts, Inc.
PROBES
![Page 58: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/58.jpg)
58Prepared by Professional Data Analysts, Inc.
INTERVIEW PROTOCOL
![Page 59: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/59.jpg)
59Prepared by Professional Data Analysts, Inc.
OVERCOMING BARRIERS / ENGAGING THE RESPONDENT
![Page 60: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/60.jpg)
60Prepared by Professional Data Analysts, Inc.
Formats:
Online
In-person
Sampling:
Demographics
Level of acquaintance
Ability to verbalize thoughts
(can be a screener)
Number of groups:
Convergence of information
Resources available
Focus Groups provide an interactive, facilitated session where group dynamics are important
**Group characteristics will determine how data is analyzed – groups are typically compared**
![Page 61: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/61.jpg)
61Prepared by Professional Data Analysts, Inc.
TYPOLOGY OF FOCUS GROUP QUESTIONS
Background / Icebreaker
Main evaluation questions
Factual questions
Anonymous questions
Kitchen sink questions (“Are there any other issues we haven’t discussed that you would
like to bring up?” / “Let’s discuss at the end…”)
Big picture question (“The department of health is trying to understand what type of
cessation service is most appealing to tobacco users in the state. If you had the opportunity to
take your own message to the state what would you say?”)
![Page 62: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/62.jpg)
62Prepared by Professional Data Analysts, Inc.
GROUP DYNAMICS NECESSITATE FACILITATION BY MODERATOR
![Page 63: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/63.jpg)
![Page 64: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/64.jpg)
64Prepared by Professional Data Analysts, Inc.
When to use:
When you are attempting to
understand an ongoing
process or situation
There is a need to
understand a physical
setting (e.g., schools,
classrooms)
When it is difficult to collect
data from individuals
Types:
Direct – watching a
classroom teacher
Indirect – measuring the
amount of food waste in a
school cafeteria
Drawbacks:
Can be resource intensive
Susceptible to Hawthorne
effect
On its own does not address
why behaviors, interactions,
or processes are occuring
Observation
![Page 65: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/65.jpg)
65Prepared by Professional Data Analysts, Inc.
Medical charts
Earned media
(e.g., newspaper
mentions)
Legal records
Extraction – retrieving data out of (poorly structured) data sources for further analysis
![Page 66: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/66.jpg)
66Prepared by Professional Data Analysts, Inc.
Census data
BRFSS or other
surveillance data
MN Population
Center data
Secondary data sources
![Page 67: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/67.jpg)
67Prepared by Professional Data Analysts, Inc.
Geographic Information
Systems (GIS)
Non-verbal communication
Photographs, video (You
Tube, self-cam, others)
Pictures, writing, work
samples, etc.
Innovative data collection
![Page 68: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/68.jpg)
68Prepared by Professional Data Analysts, Inc.
NEXT STEPS: PUTTING IT INTO ACTIONData Collection Methods:
1. Mailed survey
2. Online survey
3. Individual, face-to-face interviews
4. Focus group interview
5. Phone survey or interview
6. Observation
7. Archival data (records and documents)
8. Test
Minnesota Evaluation Studies Institute Spring Training 2017
![Page 69: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/69.jpg)
Section 3: Sampling
![Page 70: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/70.jpg)
70Prepared by Professional Data Analysts, Inc.
THE IMPORTANCE OF SAMPLING
![Page 71: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/71.jpg)
71Prepared by Professional Data Analysts, Inc.
DEFINE YOUR SAMPLE
Be informed about and sensitive to needs and constraints of your sample
How to best reach a homeless population? How to best reach young adults? Are there cultural considerations (e.g., gender, ethnicity, geography, etc.)?
![Page 72: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/72.jpg)
72Prepared by Professional Data Analysts, Inc.
HOW LARGE SHOULD THE SAMPLE BE?
It Depends!
- Do you need to generalize to a broader population?
- At what point will I have reached saturation?
- What is feasible?
- What is ethical?
- Which subgroups need to be represented?
- Which subgroups may be over-represented?
Minnesota Evaluation Studies Institute Spring Training 2017
![Page 73: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/73.jpg)
73Prepared by Professional Data Analysts, Inc.
Random
Stratified
Structured (purposeful
oversampling)
Cluster (specific group –
daycare settings, school, etc.)
Judgment/experts
Snowball sampling
Typical case
Extreme or deviant cases
Theory based
Maximum variation
Critical case
Success case method
Homogeneous
Confirming or disconfirming
cases
Intensity
MQP qualitative methods – 40+
types of sampling for qualitative
studies
…
Convenience
A Smattering of Sampling Strategies
![Page 74: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/74.jpg)
74Prepared by Professional Data Analysts, Inc.
ATTRITION – THE CONCEPT
![Page 75: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/75.jpg)
75Prepared by Professional Data Analysts, Inc.
ATTRITION – AN EXAMPLEStart with
N = 3950 records
Keep records where there is at least 1 field from the survey completed
Leaves n = 3695
Keep most complete unique recordLeaves n = 3629
Verify visit dates and visit types are in correct order for those with multiple
visitsLeaves n = 3628
Clean records with birthdate - visit type mismatches & remove obvious data entry
errors based on these fieldsLeaves n = 3684
Removed n = 255 records where all survey data was missing
Identified n=480 records with severe birthdate-visit type mismatches Removed n = 11 records with
obvious data entry errors
Removed total of n = 55 records (n = 4 were exact duplicates and n = 51 were duplicate on
PatientID-VisitDate-VisitType-IsPostVisit)
Removed 1 record where 1st record was 15 month visit & 2nd visit was newborn. (rest of the data support 15 month
visit, so not sure why there was also a newborn survey after the 15 month); There were 13 people (45 records) with
repeated visit types for different visit dates. These were left in as is.
Match pre- and post- visit surveysLeaves n = 1403 complete records
Of the 3628 total records, 2206 were pre-visit & 1422 were post-visit. When combined on PatientID-VisitDate-VisitType, there were 1403 complete
surveys (pre- and post-visit), 803 pre-visit only, and 19 post-visit only.
![Page 76: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/76.jpg)
Section 4: Technologies for Data Collection
![Page 77: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/77.jpg)
77Prepared by Professional Data Analysts, Inc.
Examples of Data Collection Technologies
![Page 78: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/78.jpg)
78Prepared by Professional Data Analysts, Inc.
QUALTRICS
o Online survey software, fairly
flexible
o Surveys can be customized,
there are also templates
o SMS survey invites
o Management of respondents,
reminders, completes
o Some analysis capabilities
(be careful with this though!)
o Good support, resources,
online information
(https://www.qualtrics.com/s
upport/)
Minnesota Evaluation Studies Institute Spring Training 2017
![Page 79: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/79.jpg)
79Prepared by Professional Data Analysts, Inc.
POLL EVERYWHERE
o Real-time polling ability
o Supports closed-ended or
open-ended items
o Phone, web, or Twitter
options
o One-time use available
o Limited options for free
account
Minnesota Evaluation Studies Institute Spring Training 2017
https://vimeo.com/polleverywhere/intro
![Page 80: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/80.jpg)
80Prepared by Professional Data Analysts, Inc.
V-ROOM
o Virtual qualitative and
collaboration suite
o Supports online focus
groups
o Ability to share images,
video, hyperlinks, etc. during
the focus group
o Ability to split participants
into subgroups
o Integration with web cam
o Up to six participants at a
time (excluding moderators)
Minnesota Evaluation Studies Institute Spring Training 2017
![Page 81: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/81.jpg)
81Prepared by Professional Data Analysts, Inc.
WHAT IS YOUR FAVORITE TECHNOLOGY?
o Survey Monkey
o Microsoft 2016
o Moodle
o Dragon
o Voxco
o Other??
Minnesota Evaluation Studies Institute Spring Training 2017
![Page 82: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/82.jpg)
Section 5: Pulling the Plan Together
![Page 83: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/83.jpg)
83Prepared by Professional Data Analysts, Inc.
PLANNING DATA COLLECTION DESIGN AND CONTENT
![Page 84: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/84.jpg)
84Prepared by Professional Data Analysts, Inc.
![Page 85: “The Proof is in the Pudding, the Perils are in the Plan”Data Collection Methods “The Proof is in the Pudding, the Perils are in the Plan” Melissa Chapman Haynes, Ph.D. Senior](https://reader034.vdocuments.net/reader034/viewer/2022052012/6028cd1ac4af680f6f775baa/html5/thumbnails/85.jpg)
85Prepared by Professional Data Analysts, Inc.
Questions?