1
Gathering SoTL Gathering SoTL Evidence: Methods for Evidence: Methods for Systematic Inquiry into Systematic Inquiry into
Student LearningStudent LearningRenee A. MeyersRenee A. Meyers
Coordinator, UWS SoTL Coordinator, UWS SoTL Leadership SiteLeadership Site
Faculty College, May 2009Faculty College, May 2009
2
Using Questions to Using Questions to Gather SoTL EvidenceGather SoTL Evidence
Types of Questions Types of Questions Question DesignQuestion Design Question Issues and ProblemsQuestion Issues and Problems Types of Response ScalesTypes of Response Scales Design and Preparation of SurveyDesign and Preparation of Survey
4
Methods for Gathering Methods for Gathering EvidenceEvidence
Evidence can include:Evidence can include:
Course-Intrinsic EvidenceCourse-Intrinsic Evidence
Course-Extrinsic EvidenceCourse-Extrinsic Evidence
Using your research question, what Using your research question, what is one type of course-intrinsic and is one type of course-intrinsic and one type of course-extrinsic evidence one type of course-extrinsic evidence you might collect?you might collect?
5
Asking QuestionsAsking Questions
All SoTL evidence gathering is All SoTL evidence gathering is focused on asking focused on asking questionsquestions of of students.students.
Research on writing Research on writing surveysurvey questions questions provides comprehensive information on provides comprehensive information on how to do this besthow to do this best
6
Survey EvaluationSurvey Evaluation
Look at survey in packet. Look at survey in packet. What works?What works? What doesn’t work? What doesn’t work? What is missing?What is missing?
7
What Survey Research Tells What Survey Research Tells Us About Asking Good Us About Asking Good
QuestionsQuestions
8
The Impact of Questions on The Impact of Questions on the Research Processthe Research Process
Poor questions lead to lousy results.Poor questions lead to lousy results.
How might answers to these questions How might answers to these questions differ? differ? Do you attend class?Do you attend class? Do you Do you regularlyregularly attend class? attend class? How many times per week do you attend this How many times per week do you attend this
class?class?
Asking survey questions is part science Asking survey questions is part science and part art.and part art.
9
Types of Survey Types of Survey QuestionsQuestions
Open Ended and Closed Ended Open Ended and Closed Ended Question FormatsQuestion Formats
Attitude, Knowledge, and Behavior Attitude, Knowledge, and Behavior Question FormatsQuestion Formats
10
Open-Ended QuestionsOpen-Ended Questions Open-ended questions ask respondents Open-ended questions ask respondents
to state in their own words their to state in their own words their response to the question.response to the question.
Closed-ended (or fixed alternative) Closed-ended (or fixed alternative) questions ask respondents to respond questions ask respondents to respond by picking one or more alternatives by picking one or more alternatives from a list of available options.from a list of available options.
Can you find any examples of open-Can you find any examples of open-ended questions on the survey? ended questions on the survey? Are these good open-ended Are these good open-ended questions? questions?
11
Open Ended QuestionsOpen Ended Questions
What are advantages and What are advantages and disadvantages to open- and close-disadvantages to open- and close-ended questions? ended questions?
12
Advantages and Disadvantages Advantages and Disadvantages
Are often less time consuming Are often less time consuming May provide more complete answersMay provide more complete answers May provide information May provide information
researchers did not anticipateresearchers did not anticipate Responses are often easier to Responses are often easier to
analyzeanalyze May produce better response rateMay produce better response rate
13
Surveys: Asking QuestionsSurveys: Asking Questions
Surveys are used most often in SoTL Surveys are used most often in SoTL research when we are interested in research when we are interested in asking questions about: asking questions about: how students FEEL or THINK about how students FEEL or THINK about
certain issues, topics, activities, related certain issues, topics, activities, related to their learning,to their learning,
what students KNOW about various what students KNOW about various topics, etc.topics, etc.
how often students engage in certain how often students engage in certain learning BEHAVIORSlearning BEHAVIORS
14
Attitude/Feelings QuestionsAttitude/Feelings Questions
Attitude/feelingsAttitude/feelings questions ask questions ask respondents for their views on respondents for their views on certain topics or issuescertain topics or issues There are no right or wrong answers to There are no right or wrong answers to
attitude/feeling questions.attitude/feeling questions. Attitude/feeling questions should be Attitude/feeling questions should be
“neutrally” written so that the “neutrally” written so that the respondent is not led to believe that a respondent is not led to believe that a certain response is desired.certain response is desired.
15
Knowledge QuestionsKnowledge Questions
KnowledgeKnowledge questions ask about what questions ask about what a respondent knows about a a respondent knows about a particular topicparticular topic
Knowledge questions can be asked as Knowledge questions can be asked as a screening question to determine if a screening question to determine if respondents are qualified to answer respondents are qualified to answer attitude questions.attitude questions. ““What do you know about evolution?What do you know about evolution?
“ “How do you feel about evolution as a How do you feel about evolution as a theory of creation?”theory of creation?”
16
Behavior QuestionsBehavior Questions
BehaviorBehavior questions ask respondents to questions ask respondents to report on their past or future behaviorreport on their past or future behavior Responses to behavior questions may Responses to behavior questions may
be biased because of faulty or biased be biased because of faulty or biased recall by respondentsrecall by respondents
17
Attitude, Knowledge, and Attitude, Knowledge, and Behavior QuestionsBehavior Questions
Can you find an attitude/feelings, Can you find an attitude/feelings, knowledge, and behavior question on the knowledge, and behavior question on the survey?survey?
Write one attitude/feelings, knowledge, or Write one attitude/feelings, knowledge, or behavior question for your own SoTL behavior question for your own SoTL project. project. Pre-assessment measurePre-assessment measure Question for an assignment you will collectQuestion for an assignment you will collect Post-assessment Post-assessment
19
Recall QuestionsRecall Questions
Is there a question on survey that Is there a question on survey that asks students to “recall” some asks students to “recall” some behavior? behavior? How would you evaluate these How would you evaluate these
questions?questions? Would you know how to answer Would you know how to answer
them? them? Is there a better way to ask these Is there a better way to ask these
questions?questions?
20
Using Diaries for Recall Using Diaries for Recall QuestionsQuestions
To minimize recall errors, researchers To minimize recall errors, researchers can ask respondents to keep personal can ask respondents to keep personal diaries to record their actual teaching diaries to record their actual teaching or learning activities over a period of or learning activities over a period of timetime Caution: Respondents may forget to fill Caution: Respondents may forget to fill
out their diaries on a timely basis and out their diaries on a timely basis and instead “catch up” their diaries just instead “catch up” their diaries just before they are due to be collectedbefore they are due to be collected
21
Unaided vs. Aided Recall Unaided vs. Aided Recall QuestionsQuestions
Because respondents often forget, or fail to recall Because respondents often forget, or fail to recall accurately, their activities and behavior, you can use accurately, their activities and behavior, you can use aided recall questions to “jog” their memories.aided recall questions to “jog” their memories. Unaided recall: “Please list all of the Unaided recall: “Please list all of the
Communication classes in which you participated Communication classes in which you participated in learning groups last semester.” (open-ended in learning groups last semester.” (open-ended question)question)
Aided recall: “Please place a check by any of the Aided recall: “Please place a check by any of the Communication courses in which you participated Communication courses in which you participated in a learning group last semester.” (closed-ended in a learning group last semester.” (closed-ended question)question)
Public SpeakingPublic Speaking Introduction to Interpersonal CommunicationIntroduction to Interpersonal Communication Small Group CommunicationSmall Group Communication
22
Recall Questions and Time Recall Questions and Time FrameFrame
We often want respondents to focus on a We often want respondents to focus on a specific time frame when answering a specific time frame when answering a question.question. Avoid ambiguous terms like “past year” or Avoid ambiguous terms like “past year” or
“next year” because respondents may not “next year” because respondents may not know if this is a calendar year (i.e., 2006, know if this is a calendar year (i.e., 2006, 2007, 2008) or a 12-month period.2007, 2008) or a 12-month period.
Avoid time frames that are too large (for the Avoid time frames that are too large (for the recall question) or too far in the past.recall question) or too far in the past.
Write one recall question for gathering Write one recall question for gathering evidence for your SoTL research question. evidence for your SoTL research question.
23
Intimidating or Intimidating or Inappropriate QuestionsInappropriate Questions
Respondents will consider a question to Respondents will consider a question to be intimidating or inappropriate if:be intimidating or inappropriate if: It addresses highly personal and private It addresses highly personal and private
behavior (drinking behavior, sexual behavior (drinking behavior, sexual activity, illegal drug use, etc.)activity, illegal drug use, etc.)
It addresses subjects for which the It addresses subjects for which the respondent believes there are socially respondent believes there are socially desirable norms (communication with desirable norms (communication with instructors, behavior in class, spiritual instructors, behavior in class, spiritual behavior, etc.)behavior, etc.)
If respondents perceive a question to be If respondents perceive a question to be intimidating, they are more likely to intimidating, they are more likely to refuse to answer or to give an inaccurate refuse to answer or to give an inaccurate answer.answer.
24
Intimidating QuestionsIntimidating Questions
Would you use an open- or Would you use an open- or closed-ended question to ask a closed-ended question to ask a potentially intimidating question potentially intimidating question to gather evidence from your to gather evidence from your students?students? Examples? Examples?
25
Forms for Asking Potentially Forms for Asking Potentially Intimidating Questions About Intimidating Questions About
BehaviorBehavior If you ask a closed-ended question If you ask a closed-ended question
with a set of response options, with a set of response options, people typically do not choose the people typically do not choose the extreme options.extreme options.
Therefore, it is often better to use Therefore, it is often better to use an open-ended question and allow an open-ended question and allow the respondents to indicate how the respondents to indicate how often they engage in certain often they engage in certain behaviors, or define the terms as behaviors, or define the terms as they perceive them.they perceive them.
26
Forms for Asking Potentially Forms for Asking Potentially Intimidating Questions About Intimidating Questions About
BehaviorBehavior Third person questions ask Third person questions ask
respondents to report on the respondents to report on the behavior and activities of other behavior and activities of other people.people. Example: Instead of asking college Example: Instead of asking college
students directly about their alcohol students directly about their alcohol consumption behavior and its impact on consumption behavior and its impact on their learning, you could ask students to their learning, you could ask students to report how much alcohol their report how much alcohol their roommates consume and how that roommates consume and how that affects their learning behavior.affects their learning behavior.
27
Double-Barreled QuestionsDouble-Barreled Questions
Double-barreled questions ask Double-barreled questions ask respondents for their opinions about respondents for their opinions about two distinct issues in the same two distinct issues in the same question. question. ““Do you prefer working in groups and Do you prefer working in groups and
learning from your fellow students, or learning from your fellow students, or do you prefer lectures and learning do you prefer lectures and learning from the instructor?from the instructor?
Any other examples?Any other examples?
28
Leading and Loaded Leading and Loaded QuestionsQuestions
Some “surveys” seek to generate Some “surveys” seek to generate results that support a set of results that support a set of predetermined goals and predetermined goals and objectives.objectives.
How might we re-write a How might we re-write a question on the survey to make it question on the survey to make it a leading question?a leading question?
30
Ordering of QuestionsOrdering of Questions Start with easy, salient, and non-Start with easy, salient, and non-
threatening questions first.threatening questions first. Complete questions on a single topic Complete questions on a single topic
before moving on to a new topic.before moving on to a new topic. When switching topics, use transitional When switching topics, use transitional
phrases to make it easier for phrases to make it easier for respondents to switch their train of respondents to switch their train of thought.thought.
Put demographic questions near the Put demographic questions near the end.end.
Do a quick evaluation of survey to see Do a quick evaluation of survey to see how it fits criteria.how it fits criteria.
31
Funnel QuestionsFunnel Questions Questions regarding a specific topic should usually Questions regarding a specific topic should usually
start with the most general questions and then start with the most general questions and then follow with increasingly more restrictive questions.follow with increasingly more restrictive questions. If you were required to work in groups in your If you were required to work in groups in your
class, what types of students would you want in class, what types of students would you want in your group?your group?
If you had to select just three key If you had to select just three key characteristics that you would use to choose characteristics that you would use to choose among these different students, what would among these different students, what would these 3 factors be?these 3 factors be?
Would “student’s GPA” be an important factor in Would “student’s GPA” be an important factor in your choice of students for your work group?your choice of students for your work group?
If your instructor indicated that she was going If your instructor indicated that she was going to put students in groups according to similar to put students in groups according to similar GPAs, would you be interested in working in GPAs, would you be interested in working in such a group?such a group?
32
Skipping and BranchingSkipping and Branching Surveys can instruct respondents Surveys can instruct respondents
to skip over certain questions if to skip over certain questions if these questions do not pertain to these questions do not pertain to them.them.
Question 7: “Have you worked in a Question 7: “Have you worked in a classroom learning group in the classroom learning group in the past semester?”past semester?” ____ Yes (please proceed to question 8)____ Yes (please proceed to question 8) ____ No (please skip to question 12)____ No (please skip to question 12)
Any examples on survey?Any examples on survey?
33
Demographic QuestionsDemographic Questions In most surveys, questions regarding In most surveys, questions regarding
demographic characteristics of the demographic characteristics of the respondents are included at end.respondents are included at end.
Typical demographic questions:Typical demographic questions: SexSex Ethnicity/Racial backgroundEthnicity/Racial background AgeAge Year in SchoolYear in School MajorMajor
Are there better ways to write these Are there better ways to write these demographic questions on the survey?demographic questions on the survey?
35
Avoid Ambiguity in Avoid Ambiguity in WordingWording
Some common words, such as “regularly” or Some common words, such as “regularly” or “often”, may be interpreted very differently by “often”, may be interpreted very differently by respondents.respondents. Do you go to talk to your instructor regularly Do you go to talk to your instructor regularly
(outside of class)?(outside of class)? How often do you go talk to your instructor (outside How often do you go talk to your instructor (outside
of class):of class): 1-2 times a week1-2 times a week 3-4 times a week3-4 times a week 5-7 times a week5-7 times a week More than 7 times a weekMore than 7 times a week
What might be the issue with these What might be the issue with these questions? questions?
37
Features of Response Features of Response CategoriesCategories
Any response category must be Any response category must be mutually exclusivemutually exclusive.. There should not be any overlap in the There should not be any overlap in the
categories.categories. Any response category Any response category systemsystem should be should be
collectively exhaustivecollectively exhaustive.. All possible responses should be included in the All possible responses should be included in the
category category systemsystem.. Usually it is advisable to include an “other” Usually it is advisable to include an “other”
category to allow respondents to add a response category to allow respondents to add a response that is not included in the category system.that is not included in the category system.
On survey, evaluate questions for exclusivity On survey, evaluate questions for exclusivity and collective exhaustion. Any issues? and collective exhaustion. Any issues?
38
Semantic Differential Semantic Differential ScalesScales
A scale, typically used to measure A scale, typically used to measure attitudes or perceptions, that includes attitudes or perceptions, that includes end points that are anchored by bipolar end points that are anchored by bipolar adjectives.adjectives.
Outstanding _ _ _ _ _ _ _ Outstanding _ _ _ _ _ _ _ UnacceptableUnacceptable
StrongStrong _ _ _ _ _ _ _ _ _ _ _ _ _ _WeakWeakMoralMoral _ _ _ _ _ _ _ _ _ _ _ _ _ _ ImmoralImmoral
Any examples of a semantic differential Any examples of a semantic differential scale on survey? scale on survey?
39
Likert ScalesLikert Scales A scale, typically used to measure attitudes, that A scale, typically used to measure attitudes, that
usually includes end points of “strongly agree” and usually includes end points of “strongly agree” and “strongly disagree.”“strongly disagree.”
Abiding by page limits is important for writing a good Abiding by page limits is important for writing a good paper:paper:
Strongly AgreeStrongly AgreeAgreeAgreeNeutralNeutralDisagreeDisagreeStrongly DisagreeStrongly Disagree
Any examples of a Likert scale on the survey?Any examples of a Likert scale on the survey?
40
Numerical ScalesNumerical Scales A scale that uses multiple numbers, usually A scale that uses multiple numbers, usually
5 to 10, with anchored end points to 5 to 10, with anchored end points to represent the strength or position of represent the strength or position of respondents’ attitudes. respondents’ attitudes.
1 strongly disagree1 strongly disagree 2 disagree2 disagree 3 neutral3 neutral 4 agree4 agree 5 strongly agree5 strongly agree
Any examples of a numerical scale on Any examples of a numerical scale on survey?survey?
41
The Use of “Don’t Know” and “Not The Use of “Don’t Know” and “Not Sure” and “Not Applicable”Sure” and “Not Applicable”
Rather than have respondents feel Rather than have respondents feel forced to give an invalid response, forced to give an invalid response, researchers often add a category to researchers often add a category to scales that is labeled “don’t know” scales that is labeled “don’t know” or “not sure” or “not applicable”.or “not sure” or “not applicable”.
Write a question for your research Write a question for your research study that includes a response scale. study that includes a response scale. Then explain why you wrote the Then explain why you wrote the question using that response scale. question using that response scale.
44
Length of SurveysLength of Surveys
How long respondents will spend How long respondents will spend answering a survey is directly related answering a survey is directly related to the saliency of the issues for them.to the saliency of the issues for them.
For salient topics, questionnaires of For salient topics, questionnaires of 10-15 pages are possible.10-15 pages are possible.
For non-salient topics, questionnaires For non-salient topics, questionnaires are usually limited to 2 to 4 pages.are usually limited to 2 to 4 pages.
How does this survey rate on these How does this survey rate on these criteria?criteria?
45
Introducing the SurveyIntroducing the Survey In a cover letter or introductory In a cover letter or introductory
comments, the following points comments, the following points should be addressed:should be addressed: What the study is about and its relevance to What the study is about and its relevance to
the participant;the participant; Why the respondent’s cooperation is Why the respondent’s cooperation is
important;important; Promise of confidentiality and security of Promise of confidentiality and security of
information (IRB rules and regulations must information (IRB rules and regulations must be followed);be followed);
Reward for participation (if relevant);Reward for participation (if relevant); Deadline for completion (if relevant);Deadline for completion (if relevant); Who to contact regarding any questions;Who to contact regarding any questions; Thank youThank you
Evaluate survey in terms of these criteriaEvaluate survey in terms of these criteria
46
Pretest, pretest, pretestPretest, pretest, pretest
You should always pretest draft versions You should always pretest draft versions of your questions with representatives of your questions with representatives from the target population.from the target population.
But never pretest your questionnaire with But never pretest your questionnaire with people who may be in your final sample of people who may be in your final sample of respondents.respondents.
Who might I ask to pretest this survey? Who might I ask to pretest this survey?
47
Online survey help sitesOnline survey help sites
Advanced SurveyAdvanced Survey ZoomerangZoomerang Web Online SurveysWeb Online Surveys SurveyMonkeySurveyMonkey
48
Final Comments on Survey Final Comments on Survey DesignDesign
Never underestimate the time to takes to Never underestimate the time to takes to create a high quality survey instrument.create a high quality survey instrument.
No survey is ever perfect.No survey is ever perfect. There is no substitute for pre-testing and There is no substitute for pre-testing and
soliciting feedback from other research soliciting feedback from other research experts.experts.
Shorter is always better.Shorter is always better. Only ask questions on surveys that are Only ask questions on surveys that are
directly related to specific and important directly related to specific and important research issues.research issues.
49
ReflectionReflection
How might I use questions to collect How might I use questions to collect evidence for my research project?evidence for my research project?
What will be most important for me What will be most important for me to remember about the questions I to remember about the questions I want to ask?want to ask?