closing the loop: assessing lmu’s undergraduate learning outcomes

Closing the Loop:Assessing LMU’s Undergraduate

Learning Outcomes

Laura J. Massa, Ph.D.Director of Assessment

1

Please Note:

Please note: The data provided in the following presentation are provided for the purposes of internal assessment and improvement. Findings should not be communicated to an external audience (e.g., website, bulletin or other publicity materials) without the expressed authorization of the Office of Assessment.

2

Outline

• Goals of the Presentation

• Undergraduate Learning Outcomes

• Closing the Loop

• Sources of Evidence

• Focus on 3 Learning Outcomes

• Additional topics of interest from NSSE

• What’s next?

3

Goals of the Presentation

LMU places a high value on teaching and learning, on providing a transformative educational experience, and on inquiry processes that focus on understanding and improving student learning.

–Guiding Principles for Assessment at LMU

1. Provide evidence of student achievement of selected outcomes

2. Engage you in process of closing the loop for the assessment of selected outcomes

4

Undergraduate Learning Outcomes

• Adopted in February 2010• 2 ½ year development process• Endorsed by the Faculty Senate

• Outcomes can be found at www.lmu.edu/assessment

• Focusing on selected outcomes:• Creative and Critical Thinking• Respect for Others• Written and Oral Communication

5

http://www.lmu.edu/assessment

Closing the LoopAssessment: A systematic, ongoing process aimed at understanding and improving student learning

ARTICULATEMISSION/

GOALS

IDENTIFYSPECIFIC

OUTCOMES

DETERMINEPRACTICES USED

TO ACHIEVEOUTCOMES

GATHEREVIDENCE

REVIEW &INTERPRETRESULTS

RECOMMENDACTIONS

6

Sources of Evidence

• Direct Measure: Looks at student work products or performances that demonstrate learning

• Rubric to assess Written Communication

– Papers collected in Spring 2010

• Indirect Measure: Captures students’ perceptions of their learning and the educational environment that supports learning

• National Survey of Student Engagement (NSSE)

– Participated in Spring 2010

7

NSSE• Assesses the extent to which students engage in educational

practices associated with high levels of learning and development

• NSSE items + Jesuit Consortium items align with our University Undergraduate Learning Outcomes

• All LMU Freshmen and Seniors invited to participate

LMU Jesuit Masters NSSE Total

# Institutions 1 11 122 563

# Respondents 856 8,935 96,847 360,611

Response Rate 35% 37% 31% 32%

8

Creative and Critical Thinking

Students will be able to ask questions, solve problems

and produce works through the innovation of ideas and

concepts and by developing and justifying solutions

through critical evaluation and analysis

9


LMU Jesuit Masters NSSE Total30%

40%

50%

60%

70%

80%

90%

100%

90% 89%

82% 83%

92% 92%87% 87%

To what extent has your experience at this institution contributed to your thinking critically and analytically?

1= very little, 4 = very much First YearSeniors

Pe

rce

nt

ind

ica

tin

g "

Ve

ry m

uc

h"

or

"Qu

ite

a b

it"

Based on NSSE data. Values in red refer to significant differences between mean responses of LMU students and corresponding students (p<.05).

10



40%

50%

60%

70%

80%

90%

100%

58%

69%

59% 59%

72% 71%

63% 64%

To what extent has your experience at this institution contributed to your ability to solve complex real-world

problems?1= very little, 4 = very much

First Year

Seniors

Pe

rce

nt

Ind

ica

tin

g "

Ve

ry m

uc

h"

or

"Qu

ite

a b

it"

11


Creative and Critical ThinkingHow much has your coursework emphasized:

Mean Response (1-4 scale; 1 = very little, 4 = very much)

LMU Jesuit MastersNSSETotal

Analyzing the basic elements of an idea, experience, or theory, such as examining a particular case or situation in depth and considering its components

FY 3.37 3.33 3.10 3.15

SR 3.46 3.43 3.26 3.29

Making judgments about the value of information, arguments, or methods, such as examining how others gathered and interpreted data and assessing the soundness of their conclusions

FY 3.02 3.08 2.95 2.94

SR 3.11 3.17 3.04 3.05

Applying theories or concepts to practical problems or in new situations

FY 3.17 3.21 3.05 3.08

SR 3.34 3.38 3.24 3.25

Based on NSSE data. Means in red are significantly different from the mean responses of LMU students (p<.05)

12

Respect for Others

Students will demonstrate respect for individual and

group difference in their interactions with others

13

Respect for Others


LMU Jesuit Masters NSSE Total1

1.5

2

2.5

3

3.5

4

2.85 2.93

2.72 2.742.83

2.742.57 2.57

To what extent does your institution emphasize en-couraging contact among students from different eco-

nomic, social, and racial or ethnic backgrounds?1= very little, 4 = very much First Year

Seniors

Me

an

Re

sp

on

se

(1

-4 s

ca

le)

14

Respect for Others


During the current school year, about how often have you done each of the following:

Mean Response (1-4 scale; 1 = never, 4 = very often)


Had serious conversations with students of a different race or ethnicity than your own

FY 2.86 2.73 2.57 2.61

SR 3.06 2.79 2.66 2.68

Had serious conversations with students who are very different from you in terms of their religious beliefs, political opinions, or personal values

FY 2.87 2.84 2.65 2.69

SR 2.99 2.87 2.69 2.72

15

Respect for Others

To what extent has your experience at this institution contributed to your development in each of the following areas:

Mean Response (1-4 scale; 1 = very little, 4 = very much)

LMU Jesuit

Demonstrating respect for others’ differences FY 3.16 3.22

SR 3.28 3.19

Actively working toward a more inclusive community FY 2.92 2.92

SR 2.92 2.82


16

Oral Communication

Students will effectively express information both in writing

and orally using conventions and forms appropriate to the

intended audience

17

Oral Communication


40%

50%

60%

70%

80%

90%

100%

69% 68% 69% 67%

77% 77%75% 73%

To what extent has your experience at this institution contributed to your ability to speak clearly and effec-

tively?1= very little, 4 = very much

First YearSeniors

Pe

rce

nt

ind

ica

tin

g “

Ve

ry m

uc

h”

or

“Q

uit

e a

bit

”18


Oral Communication

During the current school year, how often have you done the following:

Mean Response (1-4 scale; 1 = never, 4 = very often)


Made a class presentationFY 2.35 2.26 2.32 2.29

SR 3.07 2.89 2.86 2.79

Tutored or taught other students (paid or voluntary) FY 1.81 1.79 1.65 1.70

SR 2.18 1.92 1.81 1.86


19

Written Communication

Students will effectively express information both in writing

and orally using conventions and forms appropriate to the

intended audience

20

Rubric for Written Communication• Applied to a random sample of 75 final papers from

multiple sections of Phil 320: Ethics

• Teaching Fellows from the English Department hired and

trained to apply rubric to papers

• Each paper independently read by 2 Teaching Fellows

Student authors from:

BCLA CBA CFA CSE SFTV

28% 30.7% 14.7% 14.7% 12%

21

Accomplished4

Proficient3

Developing2

Novice1

Context & Purpose

Demonstrates a clear and effective understanding of context, audience, and purpose that is responsive to all elements of the assignment and focuses all elements of the work.

Demonstrates adequate consideration of context, audience, and purpose and a clear focus on the assignment.

Demonstrates inconsistent awareness of context, audience, purpose. May not address all elements of the assignment.

Demonstrates minimal attention to context, audience, and purpose. Does not address assignment.

Thesis/ Central Idea

Thesis/central idea is clearly communicated, worth developing, and engaging.

Presents a thesis/central idea that can be developed.

States thesis/central idea that is weak, or too broad to be developed.

Attempted thesis/central idea is unclear

Organization & Coherence

Uses a logical structure appropriate to paper’s subject, purpose, and audience Sophisticated transitional sentences often develop one idea from the previous one or identify their logical relations. It guides the reader through the chain of reasoning or progression of ideas.

Shows a progression of ideas and uses fairly sophisticated transitional devices (e.g., may move from least to more important idea).

May list ideas or arrange them randomly rather than using any evident structure. May use transitions, but they are likely to be sequential (first, second, third) rather than logic based.

May have random organization, lacking internal paragraph coherence and using few or inappropriate transitions.

Support & Development

Uses appropriate, relevant, and compelling content to support ideas, convey understanding of the topic and shape the whole work.

Content is appropriate and relevant so that ideas are supported sufficiently. Work is generally shaped through support.

Demonstrates use of supportive content but assumes that supportive content speaks for itself and needs no application to the point being discussed, or inconsistently supports ideas with content.

Often uses ineffective or inappropriate content (e.g., opinions, examples, or clichés) to support points, or offers little evidence of any kind.

Style Uses words with precise meaning and an appropriate level of specificity. Sentences are varied, yet clearly structured and carefully focused, not long and rambling.

Primarily uses words accurately and effectively. Sentences are primarily clear, well-structured, and focused, though some may be awkward or ineffective.

Word choice is sometimes vague, imprecise, or inappropriate. Sentence structure is generally correct, but sentences may be wordy, unfocused, repetitive, or confusing.

Misuses words; employs inappropriate language. Contains many awkward sentences; sentence structure is simple or monotonous.

Mechanics Almost entirely free of spelling, punctuation and grammatical errors.

May contain some errors, which may annoy the reader but not impede understanding.

Contains several mechanical errors, which may temporarily confuse the reader but not impede overall understanding

Contains either many mechanical errors or a few important errors that block the reader’s understanding and ability to see connection between thoughts.

22


Context & Purpose

Thesis/Central Idea

Organization & Coherence

Support & Development

Style Mechanics1.00

1.50

2.00

2.50

3.00

3.50

4.00

3.183333333333343.05

3.153.13333333333333 3.13.316666666666663.22

3.04 2.933.14

3.013.18

Written Communication Rubric Scores1= novice, 4 = accomplished

Juniors (N=30)

Seniors (N= 45)

Me

an

Sc

ore

(1

-4 s

ca

le)

23


Conte

xt &

Pur

pose

Thesis

/Cen

tral I

dea

Organ

izatio

n & C

oher

ence

Suppo

rt & D

evel

opm

ent

Style

Mec

hani

cs0 0 2 2 2 117 22 21 18 17 15

43 47 43

3851

47

156 9

175 12

Written Communication Rubric Scores by Frequency

4

3

2

1

24



LMU Jesuit Masters NSSE Total1

1.5

2

2.5

3

3.5

4

3.23 3.193.08 3.04

3.21 3.273.13 3.13

To what extent has your experience at this institution contributed to your ability to write clearly and effec-

tively?1 = very little, 4 = very much First Year

Seniors

Me

an

Re

sp

on

se

(1

-4 s

ca

le)

25



40%

50%

60%

70%

80%

90%

100%

76%

56%

61% 58%

41% 43%

50% 48%

During the current school year, how often did you prepare two or more drafts of a paper

before turning it in?1= never, 4 = very often

First YearSeniors

Pe

rce

nt

ind

ica

tin

g “

Ve

ry o

fte

n”

or

“Oft

en

”26


Written CommunicationDuring the current school year, how much reading and writing have you done?

1= None, 2= 1-4, 3= 5-10, 4= 11-20, 5= More than 20


Number of written papers or reports of 20 pages or more FY 1.45 1.24 1.33 1.32

SR 1.79 1.71 1.64 1.65

Number of written papers or reports between 5 and 19 pages FY 2.93 2.53 2.23 2.27

SR 2.82 2.82 2.52 2.55

Number of written papers or reports of fewer than 5 pages FY 3.50 3.31 3.02 3.03

SR 3.28 3.20 2.96 3.00Based on NSSE data. Means in red are significantly different from the mean responses of LMU students (p<.05)

27

Additional Topics

Academic Challenge

28

Academic Challenge


LMU Jesuit Masters NSSE Total1.00

1.50

2.00

2.50

3.00

3.50

4.00

2.79 2.79 2.75 2.73

2.892.80 2.82 2.79

How often have you worked harder than you thought you could to meet an instructors expectations?

1 = never, 4 = very oftenFirst YearSeniors

Me

an

Re

sp

on

se

(1

-4 s

ca

le)

29

Academic Challenge


Survey Question


About how often have you come to class without completing readings or assignments?Mean Response (1-4 scale; 1 = never, 4 = very often)

FY 2.07 2.01 1.95 1.99

SR 2.27 2.11 2.03 2.09

About how many hours do you spend in a typical 7-day week preparing for class (studying, reading, writing, doing homework, rehearsing, and other academic activities)?Percent stating 11 or more hours

FY 75% 75% 56% 60%

SR 61% 67% 57% 61%

To what extent does your institution emphasize spending significant amounts of time studying and on academic work?Mean Response (1-4 scale; 1 = very little, 4 = very much)

FY 3.25 3.28 3.12 3.16

SR 3.13 3.23 3.13 3.16

30

Academic Challenge



2.00

3.00

4.00

5.00

6.00

7.00

5.605.66

5.41 5.485.56 5.60 5.52 5.51

To what extent have your examinations during the current school year have challenged you to do your

best 1 = very little, 7 = very much First Year

Seniors

Me

an

Re

sp

on

se

(1

-7 s

ca

le)

31

Additional Topics

Student Satisfaction

32

Student Satisfaction


1.50

2.00

2.50

3.00

3.50

4.00

3.33 3.383.19 3.23

3.45 3.44

3.19 3.24

How would you evaluate your entire experience at this institution?

1 = poor, 4 = excellentFirst YearSeniors

Me

an

Re

sp

on

se

(1

-4 s

ca

le)


33

Learning Outcomes Summary• Creative and Critical Thinking

• Clear improvement from 1st to senior year• LMU contributes to ability and coursework emphasizes this

• Respect for Others• Clear improvement from 1st to senior year• LMU emphasizes this and behaviors reflect learning

• Oral Communication• Clear improvement from 1st to senior year• LMU contributes to ability and behaviors reflect skill

• Written Communication• 1st year students report writing more and being engaged in writing

more drafts of their papers than seniors• Rubric scores indicate that more students are ‘developing’ than

‘accomplished’ in each of the 6 elements examined

34

What’s next?1. Discuss evidence with faculty in your program:

• www.lmu.edu/assessment: University Assessment Reports• Program assessment data on related learning outcomes

2. Recommend actions for improvement within your program• Possible types of actions:

– Making improvements to pedagogy, assignments or curriculum

– Planning for professional development– Allocation of resources

3. Make record of actions taken

4. Report actions to Office of Assessment

35

http://www.lmu.edu/assessment

For more on assessment:

Email: [email protected]

Website: www.lmu.edu/assessment

36

closing the loop: assessing lmu’s undergraduate learning outcomes

Documents