s tudent a ffairs r esearch and a ssessment assessment on the ground what you need to know when...
Post on 27-Dec-2015
215 Views
Preview:
TRANSCRIPT
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
ASSESSMENT ON THE GROUNDWhat you need to know when conducting assessment or working with an assessment professional so that the results help you do your best work!
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
SARAMission
Student Affairs Research and Assessment (SARA) provides leadership and service to the Division of Student Affairs in the areas of assessment, learning outcomes, educational programming, and strategic planning. SARA collects and disseminates data about Penn State students, their experiences and learning, and their perceptions of the campus environment. SARA enhances students’ educational experiences through the design, implementation, and evaluation of programs that support the University’s Cocurricular Learning Outcomes.
http://studentaffairs.psu.edu/hr/pdf/intorgchart.pdf
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Why is assessment important?It will help you do a better job
It will help ensure that you have the resources to do that job
It can help you prioritize your efforts
It can contribute to the broader conversation and understanding
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
What do you need to know?Not everyone has to be an assessment
expert
But you do need to know how to make informed decisions
◦ To conduct an assessment
◦ To use the findings effectively
Plan, Do, Check, Act (PDCA)
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
You are the content expertThings to consider:
◦ The parameters of the activity you are assessing
◦ The conceptual model that underlies the activity
◦ The desired objectives/outcomes (learning or other)
◦ Past assessment efforts and findings
◦ How this activity/assessment relates to the strategic goals of your unit and student affairs
◦ What resources are available
◦ How you will use the results
◦ Where/how will you disseminate the results
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Basic Approaches (Schuh & Upcraft, 2001)1.
2.
3.
4.
5.
6.
7.
8.
9.
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Key steps - PDCA Identify objectives
Identify appropriate measures
Choose an appropriate assessment method
Choose an appropriate project design
Collect the data
Analyze the data
Disseminate the findings
Take action
Start all over again!
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Identify & articulate objectives
You should always be able to tie what you are doing (programming, advising, etc.) to a specificobjective
Bloom’s taxonomy or similar learning constructs may be useful if you’re seeking to measure learning
The focus of a learning objective is not on what you will teach or provide, but on what students will learn or gain
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
One helpful approach to writing learning outcomes is ABC(D):
Audience: Who will accomplish the objective (e.g., students)
Behavior: What is expected; most helpful when overt and observable
Condition: Under what circumstances the behavior is expected
(Degree: The acceptable performance level - often unstated)
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Avoid the “weasel words”• Learn
• Know
• Be aware of
• Be familiar with
• Have a firm grasp of
• Understand
• Appreciate
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Example using Bloom Remember – After attending the Women's’ Resource Center open
house, students will be able to describe the resources available at the Center.
Understand – After completing their first-year seminar, participants will be able to interpret university policies related to academic integrity.
Apply – By the completion of the Study Skills series, participants will establish a realistic time management plan.
Analyze – By the end of their first elected term, student government participants can find and provide credible evidence to support a planned initiative.
Evaluate – After completing Leadership 101, class members will be able to debate the merits of various leadership styles and their application in different situations.
Create – By the end of their first elected term, student leaders will be able to develop and present detailed plans for proposed projects.
Audience Behavior Condition
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Determine appropriate measuresAttitudes?
Experiences?
Knowledge?
Abilities?
Persistence?
Graduation?
Employment?
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
More on measuresNorm-referenced = report
student performance relative to other students
Criterion-referenced = compares to an absolute standard of achievement (e.g., you pass or you fail)
Self-referenced = compares different scores/ratings from the same student
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
What kind of data?Quantitative
◦Can be economical, generalizable, reliable, valid
Qualitative
◦Can provide rich data – can ask about relationships rather than infer
Mixed methods
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Consider various types of data collection approaches
Test of abilities or knowledge
Survey
Interview
Focus group
Other types
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Tests
◦Can be used to measure a specific skill or knowledge base
◦“Objective”
◦Feasible?
Surveys
◦Attitudes
◦Experiences
◦Self-reported abilities & knowledge
◦Can be confidential/anonymous
◦Can reach large numbers of people
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Interviews
◦Rich & detailed information
◦Body language can be helpful
◦Respondents may feel inhibited
◦Repeated interviews can build trust/comfort
◦Time-consuming
Focus groups
◦Many of the same benefits as interviews, but not as in-depth
◦Participants “feed off of” each other
◦Can reach more people than interviews
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Other ways to collect dataStudent self-assessments
Portfolios
Reflective journals
Observation-based
What others can you think of?
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Project designGo back to your assessment question(s)
◦ What do you want to know?
◦ What are the limitations (e.g., time, resources, staff)?
◦ How will the assessment be used?
The answer to these questions help you figure out:
◦ What your purpose/question is
◦ What approach to use
◦ What type of information to collect
◦ How to analyze your data
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Data analysisMake sure that your analysis is appropriate
to your question.
◦ Does your data meet the basic assumptions of the statistical test you are using?
◦ Is your analyst free coding your transcripts when your goal was to analyze your data using a pre-assigned coding system based on the theoretical framework underlying your program?
Make sure that you understand the assumptions and limitations of any analyses you rely on or report, whether you are the one who conducted them or not
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Reporting & DisseminationThink about your audience
Administrators don’t have time to read a dissertation or even a journal-length article
Executive summary of the executive summary!
Put key points into easily digested tables, charts, graphs, and bullets
Don’t get carried away!
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Dissemination continuedDO NOT FORGET to spell out the
implications of your work
◦How do the findings improve our understanding?
◦What should we change?
◦What else do we need to know?
Make sure the information gets to key decision makers – understanding the structure and politics of your institution is very important (Terenzini’s “contextual intelligence”)
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
You’re not done yet!Time for action.
What are you going to do with what you learned?
Plan, Do, Check, Act, Repeat
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Theory to practiceIn practice, you will not always
have the resources to do assessment the way it is outlined in books and papers
What’s important?
◦Understand the impact of the compromises you will make
◦When you act, you are doing so based on the best available information
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Your turnGet out a piece of paper
Choose a student affairs area to represent
Write at least one learning outcome for your unit
How would you assess it?
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Think aboutType of approach (needs assessment, learning
outcomes, satisfaction, etc.)
Type of measure(s) (attitude, knowledge, etc.)
Type of data (quant, qual, mixed)
How will you collect it? (focus group, survey, etc.)
What is your design? (pre-/post-test, longitudinal, etc.)
How will you analyze it? (thematic analysis, mean comparisons, etc.)
top related