developing student learing outcomes

51
Designing Effective & Measureable Student Learning Outcomes 2015 Bonner Assessment Institute Kristin Norris, IUPUI

Upload: bonner-foundation

Post on 18-Jan-2017

290 views

Category:

Education


0 download

TRANSCRIPT

Page 1: Developing Student Learing Outcomes

Designing Effective & Measureable Student Learning Outcomes

2015 Bonner Assessment InstituteKristin Norris, IUPUI

Page 2: Developing Student Learing Outcomes

Goals for Today• Build capacity to determine WHAT you need to assess, WHY, and

how does that translate into outcomes• Align student learning outcomes with course/program activities

and assessment

Page 3: Developing Student Learing Outcomes

Getting a pulse….

Page 4: Developing Student Learing Outcomes

Getting a Pulse

• On a scale of 1-10, how confident are you that….1. Have developed measurable outcomes for your course/program?2. Have created processes that provide good evidence of that learning outcome?3. Are able to assess the student learning?4. Based upon evidence, know what to do differently in your course/program in

order to increase student learning?

• Why are you/are you not confident? What is causing you to question?

Page 5: Developing Student Learing Outcomes

Why Outcomes and Assessment Matter

•Commonly asked questions•Getting beyond ‘doing good’•Intentional decision making•Strategic plan and civic mission of institutions

Page 6: Developing Student Learing Outcomes

Importance of Good Questions

•What do you need to know and why? •Refining your questions

•Your question should dictate everything (your plan, your methods, your approach, your measures)

Page 7: Developing Student Learing Outcomes

Rigor in Research (Assessment)

• Rigorous thinking• Hypothesis exploration (Starts with a good question)• Systematic approach• Builds upon prior research• Contributes to the field• Triangulation across inquiry• Creates useful, relevant knowledge

(Patton, 2012)

Page 8: Developing Student Learing Outcomes

Schoen’s Reflective Practice

•Trait of a good professional•Reflection in-action•Attentive to new knowledge•Curious, test ideas, adjust practices•Revise plans and practices - why are you doing this?

Page 9: Developing Student Learing Outcomes

Language

Page 10: Developing Student Learing Outcomes

TERMS• Measure/Metrics

• Assessment

• Evaluation

• Research

NOTE: There are no clear or “singular” definitions utilized across community engagement (CE), which makes it even more important for us to have a clear understanding of how we define our work.

Page 11: Developing Student Learing Outcomes

Measures/Metrics… it’s about the outputs; how can CE be measured or quantified?

Main Example in Higher Education:numbers or “measurements” surrounding

“performance indicators”Examples of Measurements:

• # and % of faculty/staff/students involved in CE• # of internships in the community• $$ economic impact of CE

Page 12: Developing Student Learing Outcomes

Assessment…it’s about the outcomes; what are we trying to change or impact through CE?

Main Example in Higher Education:student learning outcomes

Examples of Assessment:• What do students learn through CE experiences?• How does CE influence students’ development?• How does CE bridge “us v. them” with the community?• What skills are developed through CE experiences?

Page 13: Developing Student Learing Outcomes

Evaluation…it’s about the practice itself; is CE effective, efficient, of a certain quality, etc.?

Main Example in Higher Education: program and/or teacher evaluation

Example Evaluation Questions:• What is a high quality service-learning experience?• Are CE programs cost-effective?• How do CE programs meet their objectives (i.e. outcomes

and outputs)?

Page 14: Developing Student Learing Outcomes

Research…it’s about being more generalizable, per se

• Institutional Research• Higher Education Research• Disciplinary Research• Community-Engaged Research (CER)• Community-Based Research (CBR)

Examples of research:• Institutional: how does IUPUI do CE?• Higher Education: how does HE do CE?• Disciplinary: how does Geography do CE?• CBR: How does one get a classroom engaged in the community?• CER: How does one engage with the community in research?

Page 15: Developing Student Learing Outcomes

Outputs, Outcomes, Impact

Inputs Outputs Outcomes Impacts

All the resources put into the project to enable the delivery of outputs

All the activities undertaken and products and services delivered

The changes, benefits, learning or other effects that result from the outputs

The effect of a project at a higher or broader level, in the longer term, after a range of outcomes have been achieved

Page 16: Developing Student Learing Outcomes

Starting with Outcomes in Mind

What do you want to assess?

Page 17: Developing Student Learing Outcomes

Where do your CE assessment interests lie?

Institution

School

Department

Course

Project 1 Project 2

Program of Study

Course

Project

Center/Office

Program

Project 1

Project 2

1

2

3

4

Page 18: Developing Student Learing Outcomes
Page 19: Developing Student Learing Outcomes

Outcome Statements Should…• Describe what students should be able to demonstrate, represent, or

produce based on their learning histories;• Rely on active verbs that identify what students should be able to

demonstrate, represent, or produce over time. (Maki, 2004)• Exist at different levels:

• Lesson/Activity• Program• School/Dept/Unit/Center (multiple programs)• Institutional

• ***Should be able to map your outcomes across all of these****

Recommended Resource: http://ccoe.rbhs.rutgers.edu/forms/EffectiveUseofLearningObjectives.pdf

Page 20: Developing Student Learing Outcomes

Why are student learning outcomes important? • At the course level

• Build a foundation for your course• Define your expectations• Describe how your course is relevant to the students’ personal, academic, and/or

professional development• Keep you focused on what is important• Enable students to articulate what they are learning and have learned.

“Students will role play the problem-solving, communication, and teamwork skills necessary to effectively resolve real-world hotel management scenarios.”

Page 21: Developing Student Learing Outcomes

Why are student learning outcomes important? • At the unit/lesson level

• Build a structure for a unit or session• Focus each unit or session• Give learners a clear picture of what to expect• Provide criteria for constructing assessments• Guide the selection of learning activities• Teach learners how to be successful

“Given a problem with two unknowns, students will, in a step-by-step fashion, describe how to solve the problem.”

Page 22: Developing Student Learing Outcomes

Components of Effective Outcomes • The intended outcome clarifies what you are attempting to assess• The intended outcome is measurable• The intended outcome is useful and meaningful• One or more methods of assessment can be explicitly tied to the intended

outcome

• The CBC Method (three parts)• Condition: Under what conditions?• Behavior: What should they be able to do?• Criterion: How (well) must it be done?

Page 23: Developing Student Learing Outcomes

ActivityIdentify issues with the learning outcome and

recommend changes for improvement

Page 24: Developing Student Learing Outcomes

What recommendations would you make for improving the learning

outcome?•Outcome 1: Students have an increased ability to identify societal issues and community needs.

•Outcome 2: Students have an increased understanding of opportunities for community involvement.

•Outcome 3: Students deepen their understanding of the value of civic engagement.

•Outcome 4: Students have an increased motivation to remain civically engaged after [insert university].

•Outcome 5: Students heighten their cultural awareness and ability to work in diverse settings

Page 25: Developing Student Learing Outcomes

Self-check Questions • Is it clear what you are assessing?• Is the intended outcome measurable?

• Active verbs, active verbs, active verbs• Is the intended outcome measuring something

useful AND meaningful?• How will this outcome be measured?

Page 26: Developing Student Learing Outcomes
Page 27: Developing Student Learing Outcomes

Process for Developing Outcomes

IUPUI’s Process

Page 28: Developing Student Learing Outcomes

General Hints, Tips, and Lessons Learned

•Doesn’t happen in isolation•Iterative process•The process is just as important as the product – creates shared understanding and buy-in

•Overwhelming and potentially frustrating at first

Page 29: Developing Student Learing Outcomes

Steps Taken at IUPUI

•Regularly scheduled meetings (monthly) – all staff•Bi-weekly meetings we Dir. of Assessment and Program Director

•“Imagine you had $650,000 to develop civic-minded graduates. What would that look like”

•Post-it notes – 3-4 word statements•Group all of our ideas into categories

Page 30: Developing Student Learing Outcomes

Example from IUPUI – Civic IdentityDefinition: Students embrace their role as civic agents and as social trustees of knowledge working with others to create change in the community

Outcome:• Possess a sense of responsibility and commitment to use knowledge and skills gained through college experiences to contribute to the greater good of society(social trustee)• Make informed and principled choices that positively impact society (civic agency)• Espouse and enact values, dispositions and beliefs that lead to positive societal change • Articulate effectively how being civically engaged has fostered and informed their civic knowledge, skills, dispositions, and future intentions

Page 31: Developing Student Learing Outcomes

Example from IUPUI – Civic Communication SkillsDefinition: Students apply communication strategies to be effective in a community context, showing the ability to express, listen, and adapt ideas and messages based on the diverse perspectives of others

Outcomes:

• Listen critically to effectively respond to complex situations • Value the diverse perspectives of multiple stakeholders • Demonstrate empathy towards others• Understand the process, outcomes, and active participation in civil dialogue by applying civic communication skills • Value reciprocity and consensus building • Engage in an inclusive decision-making process that maximizes strengths and leads to mutually beneficial goals

Page 32: Developing Student Learing Outcomes

Assessment Tools and Strategies for Student

LearningPart 2

Page 33: Developing Student Learing Outcomes

Goals for Part 2

•Data considerations: What do you need, does it already exist, how will you collect it

•How will you report your findings? Who needs to know?•Example tools and strategies

Page 34: Developing Student Learing Outcomes

Context Matters When it Come to Data

Page 35: Developing Student Learing Outcomes

Activity 1

Getting to know your inner OZ

Page 36: Developing Student Learing Outcomes
Page 37: Developing Student Learing Outcomes

When you think about the context for assessing community engaged activities at your campus [within

your department, program, project, etc.], which character from the Wizard of Oz do you most identify

with and why?•Auntie Em/Uncle Henry•Cowardly Lion•Dorothy Gale•Flying Monkeys•Glinda the Good Witch•Munchkins

•Ozmites•Scarecrow•Tin Man•Toto•Wicked Witch of the West•Wizard

Page 38: Developing Student Learing Outcomes

My/Our Allies[R=real;

P=Potential]

Predators[R=real;

P=Potential]Individuals, Groups,

Policies, Systems    

  

 

My/Our Strengths[WD=Well Developed; R=Room for

Growth; U=Untapped]

Local/Global Threats

[R=Real; P=Potential]

   

ACTIVITY 2

Page 39: Developing Student Learing Outcomes

Developing an Assessment Plan

Using the Matrix

Page 40: Developing Student Learing Outcomes
Page 41: Developing Student Learing Outcomes

Completing the Matrix

• Step 1: Insert your Learning Goals and Learning Outcome/Objective (columns 1 and 2)

• Step 2: Identify course/program activities (e.g., reflections, dialogue, readings, events, trainings) – which learning outcome do they align with?

• Note: If an activity doesn’t align with one of your learning outcomes, then why are you doing it?

• Note: What happens if you’ve identified a set of program outcomes yet none of your activities can be mapped to that outcome? You have a gap in your programming

Page 42: Developing Student Learing Outcomes

Completing the Matrix (cont.)

• Step 3: Identify your sources of evidence (column 4)

• Brainstorm sources of evidence for the following examples:

• Value the diverse perspectives of multiple stakeholders • Stay abreast of contemporary issues and make informed decisions

using multiple sources of information

KEY – Be intentional and strategic

Page 43: Developing Student Learing Outcomes

Data

Page 44: Developing Student Learing Outcomes

So you say you have data…..Questions to consider:• What data do you already have?• What types of data are you collecting on a regular basis?• How are you currently using your data? Are you using all of it?• internal purposes vs. external purposes• What are you still unable to do with your data? Why?• Who decides what you collect, how you ask the questions, and how it is used?• Do you need more data? If so, what data do you need?• What lens(es) are you using when you consider these questions

Page 45: Developing Student Learing Outcomes

Sources of Evidence of Student Learning• Surveys (QR Code vs. email, timing)• Capturing the discussion during reflection sessions• Focus Group• Observations• Wordle• Second-hand reflection (in the classroom the next day)• Pictures

SYSTEMATIC

Page 46: Developing Student Learing Outcomes

Additional Considerations: Potential Factors Influencing Student Learning• Faculty interactions• Reflection • Dialogue across difference• Direct vs indirect service • Project vs hours• Student self-select partner or project vs assigned• Community partner involvement

see Whitley (2014) in MJCSL

Page 47: Developing Student Learing Outcomes
Page 48: Developing Student Learing Outcomes

Closing the Assessment Loop• If you are not using the data you are collecting – STOP

• Who needs this information and what do they care about? What is the best way to communicate/tell your story?

• Need to know vs Nice to know• Starts with a good question

Page 49: Developing Student Learing Outcomes

Examples & Tools

Page 50: Developing Student Learing Outcomes

IUPUI’s Civic-Minded Graduate

• Civic-Minded Graduate Scale (30-items)• Civic-Minded Graduate Scale Short-Form (6-items)• Civic-Minded Graduate Narrative Prompt• Civic-Minded Graduate Rubric• Civic-Minded Graduate Rubric 2.0 (see handout)• AAC&U National Task Force

Page 51: Developing Student Learing Outcomes

Already Existing Tools or Processes

•Faculty Annual Report•Campus Climate Survey•Alumni Survey•Faculty/Staff Survey•PULSE Surveys•Initiatives; Course Tags•Taxonomies