s tudent a ffairs r esearch and a ssessment to survey or not to survey? that’s a really good...
TRANSCRIPT
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
TO SURVEY OR NOT TO SURVEY? THAT’S A REALLY GOOD QUESTION!It is tempting, if the only tool you
have is a hammer, to treat everything as if it were a nail.
~ Abraham Maslow, 1966
SARA Assessment Brown BagFebruary 10, 2011
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Why is assessment important?Helps determine if you are meeting
your educational objectives
Helps ensure that you have the resources you need
Helps prioritize efforts
Can contribute to our understanding of student learning and development
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Key steps - PDCA Identify outcomes
Identify appropriate measures
Choose an appropriate assessment method
Choose and appropriate research design
Collect the data
Analyze the data
Disseminate the findings
Take action
Wash, Rinse, Repeat!
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Presentation outlineWhy do you ask?
Review types of outcomes and measures
Benefits and drawbacks to surveys
Other types of assessments
It’s your turn - practice makes perfect!
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Why not use a survey?Ease of web-based surveys has
lead to their proliferation
Result is that surveys are the “go-to” research tool, but…
◦Not every question is a nail
◦Survey fatigue is a significant threat
◦Multiple types of evidence (assessment triangulation) build a stronger case
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Starting questions1. What type of outcome do you want to
measure?
◦ Cognitive (knowledge)
◦ Affective (attitudes)
2. What type of data do you want to collect?
◦ Psychological (personal traits)
◦ Behavioral (observable activities)
3. Time frame?
◦ Short-term
◦ Long-term
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Taxonomy of Student Outcomes (Astin, 1993)
TYPE OF DATA
TYPE OF OUTCOME
Cognitive Affective
Psychological Subject-matter knowledgeAcademic abilityCritical thinking abilityBasic learning skillsSpecial aptitudesAcademic achievement
ValuesInterestsSelf-conceptAttitudesBeliefsSatisfaction with college
Behavioral Degree attainmentVocational achievementAwards or special recognition
LeadershipCitizenshipInterpersonal relationsHobbies and avocations
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Time: Examples of Short- and Long-term Outcomes (Astin, 1993)
TYPE OF OUTCOM
E
TYPE OF DATA
SHORT-TERM (DURING COLLEGE)
LONG-TERM (AFTER
COLLEGE)
Cognitive BehavioralCompletion of AlcoholEdu (vs. noncompletion)
Exhibits responsible drinking behaviors
Cognitive Psychological MCAT score
Score on medical licensing exam
Affective Behavioral Participation in student government
Involvement in local or national politics
Affective Psychological
Satisfaction with college Job satisfaction
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Types of measures
Direct measures
Indirect measures
Norm-referenced
Criterion-referenced
Self-referenced
Although direct measures are typically preferred, practically speaking, your overall assessment plan should contain a mix of these
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Your question should guide your choice of assessment tool, and not the other way around!
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Surveys
“Surveys represent one of the most common types of quantitative, social science research. In survey research, the researcher selects a sample of respondents from a population and administers a standardized questionnaire to them.”
http://writing.colostate.edu/guides/research/survey/index.cfm
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Surveys can be a good tool if you are interested in:
Perceptions
Beliefs
Motivations
Future plans
Past behavior
Private behavior
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
What about student learning?
Research indicates that aggregate self-reports of learning can provide a reasonable estimation of actual learning. *
**Self-reported data is most valid when:
◦ the information is known to the respondents,
◦ the questions are unambiguous and refer to recent activities,
◦ the respondents take the questions seriously, and
◦ responding has no adverse consequences nor does it encourage socially desirable, rather than truthful, answers.
* (Anaya, 1999; Kuh, Kinzie, Schuh, Whitt, & Associates, 2005; Laing, Swayer, & Noble, 1989; Pace, 1985; Pike, 1995). **(Kuh et al., 2005 & Pike, 1995)
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Validity and reliabilitySurveys tend to be weak on validity and strong on reliability. The artificiality of the survey format puts a strain on validity. Since people's real feelings are hard to grasp in terms of such dichotomies as "agree/disagree," "support/oppose," "like/dislike," etc., these are only approximate indicators of what we have in mind when we create the questions. Reliability, on the other hand, is a clearer matter. Survey research presents all subjects with a standardized stimulus, and so goes a long way toward eliminating unreliability in the researcher's observations. Careful wording, format, content, etc. can reduce significantly the subject's own unreliability.
http://writing.colostate.edu/guides/research/survey/index.cfm
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
RepresentativenessIf your respondents are not
representative of your population, then your results may be misleading
◦Example: You want to survey all undergraduate students about their attitudes towards a student honor code. Your friend is in charge of the FYS program and offers to have your survey passed out in all first-year seminars. You’re excited to get this direct push for your survey, but how might this effect your results?
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Strengths of surveysRelatively inexpensive
Can reach large numbers of people
Large numbers allow for multivariate analyses
Can ask many questions relatively quickly
Standardized instruments (like NSSE) allow for comparisons between groups
Can be confidential or anonymous
Can have high reliability
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Strengths continuedCan provide
student/alumni/employers perspective of the institution/program.
Can make respondents feel that their opinions matter.
Ease of response can provide information from hard to reach individuals
Results easily understood
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Weaknesses of surveysQuestions have to be general enough to
apply to all or most respondents
Inflexible - Forced-response choice may not allow respondents to express their true opinions
Require good response rates to achieve representative results
May be hard for respondents to recall information or answer truthfully
Can seldom deal with “context”
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Weaknesses continuedValidity can be questionable – results
tend to be highly dependent on wording of items, salience of survey, and organization of the instrument
Socially desirable responses
Indirect evidence which may have less legitimacy with stakeholders
Better for measuring and comparing the responses of groups rather than individuals
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Finally, it might not be a survey if….The questions you want to ask
don’t have a limited number of known, well-defined possible answers
You want to be able to ask about relationships rather than inferring them
Your population of interest differs in culture or language from the majority
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Some other types of assessments Standardized exams
Test of abilities or knowledge
Simulation or performance appraisals
Interviews and focus groups
External examiners
Archival records and transcript analysis
Portfolios
Behavior observations
Student self-evaluations
Reflective writing
Minute papers/muddiest point
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Final thoughts on choosing an assessment toolGo back to your assessment
question(s)
◦What do you want to know?
◦What are the resource limitations? (e.g., time, money, staff)
◦One shot or longitudinal?
◦Experimental design?
◦What type of analysis is appropriate?
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
NOW IT’S YOUR TURN
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Think about a program you want to assess…What do you want to know?
What type of measure?
How many students are involved?
What type of evidence do you already have (if any)?
What type of evidence is most effective with your intended audience?
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Sample assessment questionsDo participants in an alternative spring
break (n=20) develop an increased awareness of social injustice and subsequently a greater commitment to working for social justice?
Do students (n=5,000) who go through an alcohol intervention drink less as a result?
Do participants in the PRCC’s Learning Circle (n=15) exhibit improved ability to engage in positive cross-racial interaction with other participants?
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Resources Astin, A. W. (1993). Assessment for excellence: The philosophy
and practice of assessment and evaluation in higher education. American Council on Education Series on Higher Education. Phoenix, AZ: Oryx Press.
Improving Educational Programming online training module: https://www.sa.psu.edu/workshops/edprogram/index.htm
Pope, R. L., Reynolds, A. L., Mueller, J. A., & Cheatham, H.E. (2004). Multicultural competence in student affairs. San Francisco: Jossey-Bass.
SARA Assessment Resources website: http://studentaffairs.psu.edu/assessment/resources.shtml
Writing Guide: Survey Research, Colorado State University: http://writing.colostate.edu/guides/research/survey/index.cfm
Yin, A. C., & Volkwein, J. F. (2009). Assessing General Education Outcomes. In J. F. Volkwein (ed.), Assessing Student Outcomes: Why, who, what, how? New Directions for Institutional Research Assessment Supplement (pp. 79-100). San Francisco: Jossey-Bass.
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
QUESTIONS?