s tudent a ffairs r esearch and a ssessment the role of assessment in student affairs practice csa...
TRANSCRIPT
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
THE ROLE OF ASSESSMENT IN STUDENT AFFAIRS PRACTICE
CSA 502: Organization and Administration in Student AffairsFebruary 10, 2011
Building a culture of evidence:
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Presentation outlineWhat is assessment & why do we
do it?
The role of SARA and assessment offices in general
Assessment planning & the assessment cycle
It’s your turn - practice makes perfect!
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Learning objectives: At the end of this session, you should be able to:
Identify why assessment is important in all aspects of student affairs.
Describe the general role of an assessment office.
Describe the assessment cycle.
Perform the basic steps to create an assessment plan.
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
What is assessment?It is “any effort to gather, analyze,
and interpret evidence which describes…effectiveness (Upcraft & Schuh, 996)
Upcraft & Schuh (2002) argue that research informs theory while assessment informs practice.
But what does that really mean in practical terms??
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Why is assessment important?Helps determine if you are meeting
your educational objectives
Helps ensure that you have the resources you need
Helps prioritize efforts
Can contribute to our understanding of student learning and development
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Student Affairs Research & Assessment (SARA)Mission
Student Affairs Research and Assessment provides leadership and service to the Division of Student Affairs in the areas of assessment, learning outcomes, educational programming, and strategic planning. SARA collects and disseminates data about Penn State students, their experiences and learning, and their perceptions of the campus environment. SARA enhances students’ educational experiences through the design, implementation, and evaluation of programs that support the University’s Cocurricular Learning Outcomes.
http://www.sa.psu.edu/hr/pdf/intorgchart.pdf
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
What does an assessment office do?Conducts assessments, duh!
Provides training and professional development to others
Environmental scanning
Consulting
Brings together evidence
Contributes to strategic planning and accreditation activities
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Assessment planning
There is a great deal of pressure to DO assessment, but where do you start?
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
First, review your goals
Clues can be found in your:
Mission, vision, and objectives
Strategic plan
Stated learning outcomes
Where do you want to
be?
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
It is really helpful to get out of your silo
You can only learn so much from documents.
Institutional knowledge:
◦Who?
◦What?
◦How?
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Next, develop a plan.
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Start with a model
1. Janusian Duality
For internal improvement
For External accountability
II. Five Questions
1. Are you meeting your goals?
2. Are you improving?
3. Do you meet prof. standards?
4. How do you compare?
5. Are you efforts cost-effective?
III. Select Methods & Measures
For Each Level:-Institution-Unit-Individual students
For Academic Effectiveness:
Student learning,
Research, and Scholarship
For Administrative Effectiveness:
Strategic Planning &
Resource Mgmt.
IV. Collect & Analyze
EvaluationEvidence
V. Communicate
Results
Take Academic &
Administrative Action
Improve & Strengthen Programs
The Volkwein Model (2009)
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
The assessment cycle
Identify outcomes
Gather evidence
Interpret evidence
Refine efforts
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Key steps - PDCA Identify outcomes
Identify appropriate measures
Choose appropriate assessment methods
Choose an appropriate research design
Collect the data
Analyze & interpret the data
Disseminate the findings
Take action
Wash, Rinse, Repeat!
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Think long-termChances are you cannot assess
all of your outcomes every semester or even every year.
Be realistic about your resources and develop a comprehensive, multi-year plan.
Don’t forget to incorporate all steps of the cycle into your plan.
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
A good plan:Collects multiple types of evidence
(triangulation),
Takes into consideration your population(s) of interest,
Takes into consideration your audience,
Has a realistic timeline,
Identifies its products,
Incorporates dissemination & action, and
Is always subject to revision!
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
A matrix can be a helpful to organize your strategy and look for holes in your plan
Outcome Survey Reflective writing
Expert evaluatio
n
Knowledge /skills
test
Critical thinking FA2011
Leadership SP2012 FA2013
Communication FA2013 SP2014
Civic engagement SP2014
Healthy lifestyle FA2011 SP2012
Social justice perspective SP2015 FA2014
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Once you have your multi-year plan, you can bite it off in chunks.
Identify a subset of outcomes
Identify appropriate measures
Choose an appropriate assessment method
Choose an appropriate research design
Collect the data
Analyze & interpret the data
Disseminate the findings
Take action
Wash, Rinse, Repeat!
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Your turn…
Identify outcomes
Gather evidence
Interpret evidence
Refine efforts
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Example from the PRCC:By the end of the World Cultural
workshop students will be able to describe four unique traditions, values and beliefs of other races and cultures.
By the end of the social justice retreat, students will demonstrate the ability to incorporate social justice techniques.
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
AudienceBy the end of the World Cultural
workshop students will be able to describe four unique traditions, values and beliefs to other races and cultures.
By the end of the social justice retreat, students will demonstrate the ability to incorporate social justice techniques.
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
BehaviorBy the end of the World Cultural
workshop students will be able to describe four unique traditions, values and beliefs of other races and cultures.
By the end of the social justice retreat, students will demonstrate the ability to incorporate social justice techniques.
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
ConditionBy the end of the World Cultural
workshop students will be able to describe four unique traditions, values and beliefs to other races and cultures.
By the end of the social justice retreat, students will demonstrate the ability to incorporate social justice techniques.
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Next…
Identify outcomes
Gather evidence
Interpret evidence
Refine efforts
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Determine appropriate measuresAttitudes?
Experiences?
Knowledge?
Abilities?
Retention?
Graduation?
Employment?
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Some types of assessments Standardized exams
Test of abilities or knowledge
Simulation or performance appraisals
Interviews and focus groups
External examiners
Surveys
Archival records and transcript analysis
Portfolios
Behavior observations
Student self-evaluations
Reflective writing
Minute papers/muddiest point
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Tests
◦Can be used to measure a specific skill or knowledge base
◦“Objective”
◦Feasible?
Surveys
◦Attitudes
◦Experiences
◦Self-reported abilities & knowledge
◦Can be confidential/anonymous
◦Can reach large numbers of people
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Interviews
◦Rich & detailed information
◦Body language can be helpful
◦Respondents may feel inhibited
◦Repeated interviews can build trust/comfort
◦Time consuming
Focus groups
◦Many of the same benefits as interviews, but not as in-depth
◦Participants “feed off of” each other
◦Can reach more people than interviews
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Back to the PRCC: By the end of the World Cultural Workshop students will be able to describe four unique traditions, values and beliefs of other races and cultures.
Measure: Knowledge
Some possible assessment methods:
◦ Knowledge test (direct evidence)
◦ Survey self assessment of knowledge (indirect)
◦ Portfolio
◦ Reflective writing
Research design could be ex-post facto, pre-test/post-test, etc.
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Next…
Identify outcomes
Gather evidence
Interpret evidence
Refine efforts
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
What if:World Cultural Workshop
participants rated their knowledge of other cultures more highly BEFORE the workshop than after (pre-/post-test design)?
In a reflective writing exercise you notice that participants are commonly mismatching the culture/race and its unique characteristics?
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Next…
Identify outcomes
Gather evidence
Interpret evidence
Refine efforts
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Potential refinements to the World Cultural Workshop based on our assessmentHave a single assessment in
which students focus on what/how much they learned in the workshop, rather than a pre-/post-test comparison.
Cover a smaller number of races/cultures and spend more time on each.
Consider refining objectives
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
And again...
Identify outcomes
Gather evidence
Interpret evidence
Refine efforts
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Questions?If you are driving at the speed of light
and you turn on your headlights, do they do anything?
How your boss is likely to make assessment plans:
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Resources & References Astin, A. W. (1993). Assessment for excellence: The
philosophy and practice of assessment and evaluation in higher education. American Council on Education Series on Higher Education. Phoenix, AZ: Oryx Press.
Improving Educational Programming online training module: https://www.sa.psu.edu/workshops/edprogram/index.htm
Pope, R. L., Reynolds, A. L., Mueller, J. A., & Cheatham, H.E. (2004). Multicultural competence in student affairs. San Francisco: Jossey-Bass.
SARA Assessment Resources website: http://studentaffairs.psu.edu/assessment/resources.shtml
Volkwein, J. F. (2009). A model for Assessing Institutional Effectiveness. In J. F. Volkwein (ed.), Assessing Student Outcomes: Why, who, what, how? New Directions for Institutional Research Assessment Supplement (pp. 13-28). San Francisco: Jossey-Bass.