Peer Review SeminarMarquette UniversityNovember 13, 2015

Exemplary Practices at Marquette

Session One:

Exemplary Practice Presenters:

Exemplary Practice Program PAL Peer Review time

Student Learning Outcomes

Corporate Communications Sarah Feldner AM and PM

Rubrics Computer Engineering Major Susan Schneider AM Elisabeth Thompson PM

Curriculum Mapping School Counseling Masters Alan Burkard AM Accounting Masters Noreen Lephardt PM

Data Analysis Initial Teacher Licensure Program Karen Evans AM English Major Elisabeth Thompson PM Reflection English Major Sarah Wadsworth AM Accounting Major Kevin Rich PM

Student Learning OutcomesCorporate Communication Major

Student Learning Outcomes - Corporate Communication Major

Articulate the relationship between the communication function and other business functions of corporations

Explain corporate identity based on understanding of the significance of the corporate voice

Assess internal and external corporate communication needs and develop strategic communication plans

Effectively communicate with multiple stakeholder groups using various modes of communication

Articulate the role corporations play within the larger social, cultural,political and economic context of our contemporary global society.

RubricsComputer Engineering Major

Computer Engineering - Rubrics• Backstory

• 1994-1995: call for assessment of student outcomes• Preparation for NCA – HLC visit

• ABET 2000; A-K Student Learning Outcomes• Create program SLOs which map to ABET A-K, 6 main SLO for 2000 accreditation visit• 2006 accreditation visit

• ABET Criterion 3 – Student Learning Outcomes (A-K) • criteria/traits ⇔ rubrics (typical performance at 3 levels)

⇔ assessment instruments ⇔ criteria/traits

• Development Process• EECE Undergraduate Committee – research, draft, revise• EECE Faculty – review, advise, approve

• Implementation• Posted

• used with modifications by faculty in classes• used for assessment scoring, analysis, reporting

• Reviewed during assessment – wrt instrument/criteria/traits

The Electrical and Computer Engineering Department has specified 12 abilities that we expect graduates of our programs to have. They are:• (A) an ability to apply knowledge of mathematics, science and engineering• (B) an ability to design and conduct experiments as well as analyze and

interpret data• (C) an ability to design a system, component or process to meet desired needs

within realistic constraints, such as economic, environmental, social, political, ethical, health and safety, manufacturability, and sustainability

• (D) an ability to function on multidisciplinary teams• (E) an ability to identify, formulate and solve engineering problems• (F) an understanding of professional and ethical responsibility• (G) an ability to communicate effectively• (H) the broad education necessary to understand the impact of engineering

solutions in a global, economic, environmental and societal context• (I) a recognition of the need for and an ability to engage in life-long learning• (J) a knowledge of contemporary issues• (K) an ability to use the techniques, skills and modern engineering tools

necessary for engineering practice

  Level of Achievement: 1 Below Expectations

Level of Achievement: 2 Meets Expectations

Level of Achievement: 3 Exceeds Expectations

Organization

Material generally well organized, but paragraphs combine multiple thoughts or section/subsections are not identified clearly.

Organizes material in a logical sequence to enhance reader's comprehension (paragraph structure, subheadings, etc.).

Organizes material in a logical sequence to enhance reader's comprehension (paragraph structure, subheadings, etc.).

Relationship of Graphics/Figures/Tables to written material

Rarely includes GTF even when required to do so.  Does not recognize when a GTF is appropriate to organize numbers and/or facts to enhance reader understanding.

Includes any required GTF. Recognizes other opportunities to organize numbers/facts in GTF form which mimic similar organizations shown in course work (present and previous).  Explicitly discusses the GTF significance in surrounding text.

All level 2 criteria. Shows evidence of independent organization of numbers/facts in GTF form beyond any previously seen in engineering coursework.

Formatting of Graphics/Tables/Figures

GTF are sometimes difficult to read. Labels/legends/titles are missing and/or inappropriate.

GTF are easy to read.  GTF have appropriate labels and legends.  GTF have appropriate titles referring to relevant parts of the graphic (axes, legends, column/row heading, etc.)

All level 2 criteria. In addition, the graphics are visually appealing.

References Few or no references. References poorly documented if present.

Sufficient authoritative references present.  Documentation of references does not conform to any particular standard.  Most references can be traced using the information given.

Sufficient authoritative references present.All references given in IEEE reference format, or some other standard referencing scheme.  All references can be traced using the information given.

Computer Code Comments

Comments missing or inadequate for personal use by the original programmer or by another user of the code.

Comments present.  Generally well written, but missed some opportunity to provide comments of benefit to subsequent users.

Well commented code. 

G-W

: W

ritte

n C

omm

unic

atio

n

  Level of Achievement: 1 Below Expectations

Level of Achievement: 2 Meets Expectations

Level of Achievement: 3 Exceeds Expectations

Has knowledge of professional standards of ethics

Is not aware of ethical standards in EECE such as IEEE or ACM guidelines

Is aware of professional standards.

Knows, respects, and abides by professional standards.

Demonstrates ethical behavior

Has been caught cheating or plagiarizing.

Recognizes when major ethical issues are present. Does not participate in unethical activity.

Regularly incorporates ethical considerations into discussions, decision making and behavior.

Maintains professional interaction with others

Shows little respect for others, is late or absent for classes and team meetings, and makes few contributions.

Participates in team efforts, does what is required.

Shows respect for others, has professional and competent demeanor, contributes significantly to team efforts.

Demonstrates objectivity

Has personally biased perspective of problems and issues, is unable to assess things objectively.

Is able to listen to other viewpoints and tries to maintain a fair and objective perspective.

Is able to analyze a problem objectively and pro-actively recognize individual/cultural biases in themselves or others.

F: Understanding of professional and ethical responsibilities

Computer Engineering - Rubrics

• ABET Criterion 3 – Student Learning Outcomes

• criteria/traits ⇔ rubrics (typical performance at 3 levels) ⇔ assessment instruments ⇔ criteria/traits

• Development Process• EECE Undergraduate Committee –

research, draft, revise• EECE Faculty – review, advise, approve

• Implementation• Posted

• used with modifications by faculty in classes

• used for assessment scoring, analysis, reporting

• Reviewed during assessment – wrt instrument/criteria/traits

• Instrument/criteria/rubric review• F - Ethical and professional standards• I - Life Long Learning• Elevate the Standard

• Diffuse through the curriculum via rubrics

• UCCS review

• ABET • Managing the assessment process

• Consistency in interpretation, application• revision of SLO from A-K to 1-6

• Suggestions for reading<original 1994> James O. Nichols, The Departmental Guide to Implementation of Student Outcomes Assessment and Institutional Effectiveness, Agathon Press, NY 1991James O. Nichols, Karen W. Nichols, The Departmental Guide and Record Book for Student Outcomes and Institutional Effectiveness, to Implementation, 3rd ed., Agathon Press, NY 2001.Gloria Rogers, “Developing Rubrics”, ABET Webinar, March 2010. Slides available from http://apa.fiu.edu/documents_rubrics/CEC%20Rubrics/Developing%20Rubrics%20in%20Engineering%20-ABET.pdfGoogle “Rubrics for <…> Majors”; “Rubrics for <Participation, Technical Reports

Curriculum MapSchool Counseling Masters

Curriculum MapAccounting Masters

Curriculum Mapping • A process where the program faculty “map” the program's learning outcomes with the required courses in the program

• The “map” identifies in what course a student is presented the opportunity to achieve the learning outcome.

Curriculum Mapping • Curriculum mapping in a program also illustrates where the learning outcome is introduced, reinforced (developed), and mastered (applied)

• The “map” illustrates the sequencing of courses where there is an expectation of knowledge transfer for needed for more complex, applied learning.

• The accounting graduate curriculum committee developed their first curriculum map during their three year assessment cycle.

Accounting Curriculum Map

Academic Program: Masters in Accounting Program Reporting Year: 2015-16

Program Student Learning Outcomes:Explicit statements of observable, measurable results that specify what a student is expected to know or be able to do as a result of their participation in an academic program. Statements should be detailed and meaningful enough to guide decisions in program planning, improvement, pedagogy, and practice.

Map KeyI = Outcome IntroducedR = Outcome ReinforcedM = Outcome Emphasized for Mastery

Course Numbers/Program RequirementsProgram requirements can include internships, service learning, portfolios, comprehensive exams, seminars, and requirements that may not be associated with a course number.

Course Numbers/Program Requirements:

Program Student Learning Outcomes:

ACCO 5040

#1 Apply critical thinking to accounting issues

#2

#3

#4

#5

Curriculum Mapping -- The Benefits

• The mapping “process” illustrates when and where the program learning outcomes are delivered to accomplishing assurance of learning

• Supports course sequencing for students• Helps to visually represent the development of

competencies in a program• Provides the program faculty an opportunity for

reflection and curriculum evaluation in conjunction with data analysis

Data AnalysisInitial Teacher Licensure Program

edTPA Assessment Task – Formative

Evaluated by course instructorsScore of 3 is “passing”

1 2 3 4 5

IND 6a 11.1% 11.1% 50.0% 27.8% 0%

IND 6d 11.1% 11.1% 50.0% 27.8% 0%

IND 6d2 11.1% 33.3% 55.6% 0% 0%

edTPA Assessment Task – Summative

Score of 3 is “passing”

Evaluated by student teaching supervisors

1 2 3 4 5

IND 6a 8.6% 4.3% 43.5% 39.1% 4.3%

IND 6d 4.3% 17.3% 56.5% 17.3% 4.3%

IND 6d2 4.3% 30.4% 43.5% 21.7% 0%

edTPA Assessment Task – Summative

Score of 3 is “passing”

Officially scored by Pearson Scores

Average Rubric Score

IND 6a 3.0

IND 6d 2.6

IND 6d2 2.7

Student Teaching Observation

Evaluated by student teaching supervisors

KSPA 8 Does Not Meet Meets Exceeds

IND6a 00.0% 64.5% 35.5%

IND6b 00.0% 51.5% 48.4%

Exit Survey

Exit Survey Strongly Agree

Agree Disagree Strongly Disagree

6a. To use both formative and summative assessments to document learning.

44% 52% 4%

6d. To provide students with effective feedback to guide their progress toward quality work.

33% 41% 26%

Data AnalysisEnglish Major

More telling than the overall percentages cited above are the data as broken out by level. As explained above, the English Department’s new undergraduate curricula are designed around a “major sequence” consisting of three courses: a Core (UCCS) introduction to literature for non-majors as well as majors, an introduction to the discipline of English specifically designed for newly declared majors (ENGL 3000), and a capstone course to be taken in one of the final three semesters (ENGL 4997). This means that the English Department now has the ability to compare results for a single learning outcome assessed initially, at a significant milestone early in the degree program, and late in students’ progress toward the B.A. in English.

In assessing learning outcome 3 in three sections of the Introduction to the Major course (ENGL 3000) and in the one section offered of the literature Capstone (ENGL 4997), we can see a clear pattern of increased competence. Among students in ENGL 3000, 53-69% met SLO 3 and 31-40% exceeded it; in ENGL 4997, these proportions flipped, with 25% meeting the learning outcome and 67% exceeding it. Although further assessment is needed to confirm this result, this upward trend suggests a progression marked by successful teaching and learning. It also indicates that assessment within the new sequenced curricular framework is capable of providing more nuanced data than than we have had access to in the past.

Major Outcome 3 – Exceeds 2 – Meets 1 – Does Not Meet

SLO 3: Explain how history and culture inform the production and study of literature and other texts

1. ENGL 3000: Effectively analyzes primary text in context of culture, society, and history in general.

2. Capstone: Discusses primary text within context of literary history, displaying specific knowledge of literary periods and the movements, modes, and styles associated with them.

1. ENGL 3000: Adequately analyzes work in context of culture, society, and history in general.

2. Capstone: Discusses work within context of literary history, displaying general awareness of literary periods and the movements, modes, and styles associated with them.

1. ENGL 3000: Does not consider work in context of culture, society, and history in general.

2. Capstone: Does not discuss work within context of literary history.

SLO #3

Meets Expectations

Exceeds Expectations

ENGL 3000 (Introductory)

53-69% 31-40%

ENGL 4997Capstone

25% 67%

ReflectionAccounting Major

Reflection Process

• Oversight by an assessment committee (at both undergraduate and graduate levels) which meets on a yearly basis

• Examines 3-year data for each learning objective– Progress (Developing, Meets, Exceeds)– Trends– Outside factors (demographic changes, faculty

turnover, etc.)

Reflection Process

• Based on our discussion, the assessment committee formulates a list of action items, and reviews them with the department chair

• Sometimes the action is a simple as having a discussion a trend with a faculty member

• Other times we have decided on big changes in courses, such as adopting a formal writing tool for business communication

Reflection Process Example

Reflection Process Example

ReflectionEnglish Major

Assessment results for the 2014-15 academic year are encouraging. Based on the performance of 254 students across four distinct courses at three levels (Core, lower-division major, major capstone), [SW1] the percentage of students who meet or exceed the assessment criteria for the targeted learning outcomes (3 and 5) [SW2] is 93-94%.

Although these overall results are largely consistent with numbers from the past four years, it is important to recognize that direct correlations between assessment data for 2014-2015 are not possible [SW1] since this is the first year the English Department has assessed the fifteen new learning outcomes associated with the revised curriculum introduced in Fall 2014.

The English Department now has the ability to compare results for a single learning outcome assessed initially, at a significant milestone early in the degree program, and late in students’ progress toward the B.A. in English. In assessing learning outcome 3 in three sections of the Introduction to the Major course (ENGL 3000) and in the one section offered of the literature Capstone (ENGL 4997), we can see a clear pattern of increased competence. [SW1] Among students in ENGL 3000, 53-69% met SLO 3 and 31-40% exceeded it; in ENGL 4997, these proportions flipped, with 25% meeting the learning outcome and 67% exceeding it.

To monitor student learning in the new major curricula, the English Department’s Undergraduate Studies Committee is implementing, with this year’s assessment report, an updated assessment plan, which we will continue to develop over the 2015-16 academic year. Assessment results for the 2014-15 academic year were discussed by the members of the English Department Undergraduate Studies Committee on October 14, 2015. The assessment report is on the agenda for the next meeting of the department Executive Committee and will be shared with all regular faculty members in the English Department. Discussions [SW1] of learning outcomes and assessment are ongoing in the English Department.

These working groups [SW1] are significant avenues for the development, implementation, and ongoing assessment of course-specific learning outcomes linked to specific subsets of program-wide learning outcomes.

Action Steps [SW1]

Update Assessment Plan.

Complete three-year cycle focusing on two specific learning outcomes (SLO 3 and SLO 5) to obtain historical data to evaluate student learning of targeted skills and knowledge.

Begin to consider which learning outcomes to assess in the next three-year cycle, will begin in Fall 2017.

Continue conversations in pedagogy working groups regarding effective means of achieving and assessing student learning outcomes.

Follow-up on last year’s action step. [SW1]

In accordance with last year's Action Steps, the English Department assessed targeting learning outcomes associated with the new majors: ENGL (Literature), ENGW (Writing-Intensive English), and ENGA (English Language Arts). The Department is also in the process of revising its Assessment Plan.

Discussion Questions• What two overarching themes emerge from the presentations?

• Choose one or two of the presentations you saw and answer the following questions.

– What were the strengths of the assessment practice presented?

– To create and implement this practice, what did the program faculty have to think about? Discuss with each other? DO?

– How will this assessment practice likely affect student learning?

• What questions arise for you as you think about implementing best practices in your program?

• How can you adapt the practices you’ve seen to make them applicable to your program?

• What kind of support does your program need as you move forward with program assessment?

Rating the Assessment Process

Session Two:

Assessment Process Rating Guide

These criteria are intended to reflect characteristics of an effective assessment process. Are there other criteria, or different ones, that would be more appropriate for your program?

The assessment reports in ARMS will be used for the rating. Do the ARMS reports represent the true strength of your assessment? The thoughtfulness of your faculty in identifying and addressing issues related to student learning?

Would you use this guide internally? How often should a rating be done and by whom?

Select an assessment report at your table and apply the rating criteria.

Co-Curricular Learning Outcomes

Session Three:

Co-Curricular Learning Outcomes

Life SkillsAdaptive behavior that allows individuals to manage the demands of everyday

life

Holistic DevelopmentPersonal growth in mind, body and spirit that contributes to forming the whole

person

Social DevelopmentThe ability to form and manage healthy interpersonal relationships

Intercultural DevelopmentSkills and characteristics that support understanding and appropriate interaction

in a variety of cultural contexts

Social JusticeA commitment to service with and for others

Core StrategicPlan Themes

Co-Curricular

Core StrategicPlan Themes

Co-Curricular

Fostering Collaboration through Co-curricular Learning Outcomes

Are there specific classes, experiences, aspects of your program that foster these outcomes? Do you advocate for, refer or encourage your students to engage in experiences that directly relate to these outcomes? If not, what prevents you from doing that? Are there places where this information is already gathered? How could it be shared?

University Assessment Committee Members

Dr. Stephen Guastello Klingler College of Arts and Sciences 2018Dr. Marta Magiera Klingler College of Arts and Sciences 2018Dr. Pol Vandevelde Klingler College of Arts and Sciences 2017Dr. Noreen Lephardt College of Business Administration 2018Dr. Joyce Wolburg Diederich College of Communications 2016Dr. Fred Sutkiewicz School of Dentistry 2018Dr. Karen Evans College of Education 2017Dr. Baolin Wan College of Engineering 2017Dr. Guy Simoneau College of Health Sciences 2017Ms. Susan Bay Law School 2018Ms. Jean Zanoni Raynor Memorial Libraries 2017Dr. Marilyn Bratt College of Nursing 2016Dr. Maureen McAvoy College of Professional Studies 2018Dr. Jodi Blahnik Office of Student Affairs 2016Mr. Andrew Oswald Undergraduate Student: MUSG 2016Ms. Brittany Wyatt Graduate Student: GSO 2016Dr. Sharron Ronco (Chair) Assessment Director Ex-officioDr. John Su Director of Core Curriculum Ex-officioMs. Laura MacBride Institutional Research and Assessment Ex-officio