an update on morningside’s institutional effectiveness program daniel petra, edd office of...

An Update on Morningside’sInstitutional Effectiveness

Program

Daniel Petra, EdDOffice of Institutional Effectiveness

Spring 2008

What is “Institutional Effectiveness”?

●A systematic and ongoing process used by members of an institution to gather, analyze and use information to improve the institution and to monitor obtainment of its mission and goals.

●Improve: Goals, policies, procedures, plans, activities, services, functions, programs, evaluations, resource allocations, etc

Institutional Effectiveness Deals with Simple (?)

Questions●What is quality? Progress?

Achievement?●How will you know when it occurs?●How often does it occur?●What causes it?●How can you make it happen more

frequently?●How can you demonstrate it to others?

Assessment

Definitions● to measure something● an instrument used to measure something● findings or results● a process

When the Higher Learning Commission, our accreditor, uses the term "assessment" they mean the assessment of student learning; "evaluation" is used to refer to effectiveness measurement and improvement of the non-student learning functions of the institution.

At Morningside…

Assessment is defined as “the systematic and ongoing process used by faculty and students to

gather, analyze and use information to improve student learning.”

Morningside College Academic Assessment Guidelines, rev. 2005

Assessment 101

● Decisions about assessment should be guided by the Morningside College mission and vision statements.

● The purpose of assessment is to improve the educational experience of Morningside College students.

● While assessment of student learning and development should be a collaborative process involving all members of the College; responsibility for assessment rests with the faculty.

● Assessment results should be used to support curricular, planning, and budget decision making processes.

Morningside College Academic Assessment Guidelines, rev. 2005

How does IE and Assessment differ from what we already

do?

●Intentional●Explicit●Systematic●Expected

●Rewarded●Documented●Public

External Environment

Internal Factors

Institutional Effectiveness

Key Performance Indicators

Services & Functions

Student Learning

Personnel Performance

Building in Effectiveness


● The College is an open system● Approaches to institutional effectiveness

and assessment are influenced by external entities●Federal government●The Higher Learning Commission●State of Iowa●Specialized accrediting bodies●Professional organizations●Standards of best practice


Internal Factors


Internal Factors

●Mission statement●Vision statement●Board expectations●Previous experiences●Leadership●Faculty and staff participation●Student participation


Internal Factors





●Key - critical to mission, growth, and long-term survival

●Performance - institutional health, outcomes of essential activities

●Indicator – measurement, stand-alone measures or derived from secondary measures (tip of the iceberg)

Dashboard Uses

●Monitoring • institutional performance and health • impact of environmental forces and

strategic initiatives

●Performance comparisons●Problem detectors - identifying areas

for focused attention●Strategic planning●Performance management

Status of Morningside’s Dashboard

●Seven quantitative indicators●Four qualitative indicators●Approach appears to be

functioning as intended●Methods for measuring

qualitative indicators continue to mature as we gain experience and new tools


Internal Factors





Services and Functions

● Any part of the College not directly responsible for student learning (can be a gray area)

● We are all service providers● Evaluation of effectiveness may include:

● Productivity● Best practices● Service recipient satisfaction● Efficiency● Outputs (number served, number of activities,

response time, etc.)

Focus Continuum

Student Learning Focus Services/Functions Focus

•General education

•Academic majors

•Academic departments

•Student services

•Admin services

•Institutional advancement

•Finance

Status of Services and Functions Evaluation

● What is going well:● Program and office reviews have started● Some highly visible uses of student feedback

● What is going less well:● Evaluations of services and functions could be more systematic

and results more public● Systematic documentation of value gained through IE efforts

and how they are being formally incorporated into the planning and budgeting processes

● Recognition when information is acted upon and evidence of improvement

● Need to develop better methods of communicating results; too few faculty attend presentations of results

● Need a larger cadre of personnel trained in continuous improvement methods (technology analogy)


Internal Factors




Student Learning


Student Learning Assessment

Student Learning

Student Learning

Student Traits & Characteristics

● Demographics (age, gender, native language, income level, etc)

● Cognitive functioning (HS GPA, ACT scores, placement scores, etc)

● Aspirations and expectations● Self-ratings● Values and attitudes● Behavioral patterns● Educational background characteristics

Student Traits & Characteristics

● What is going well● Robust indirect assessment system in place● Longitudinal data (CIRP and CSS) is being

collected

● What is going less well● Improve distribution and use of results● Unknown whether we have any documentation

that results actually impact budgeting and planning decisions

● Unknown how well placement system is functioning


Student Learning

Environment & Experiences

●Engagement in good practices in undergraduate education

●Enriching educational experiences (includes cocurricular)

●Student-faculty interactions●Campus environment●Level of academic challenge

Environment & Experiences

● What is going well● Robust indirect assessment system in place● IDEA added another component that provides

feedback to instructors each term● Longitudinal data

● What is going less well● Improve distribution and use of results● Unknown whether we have any documentation

that results actually impact budgeting and planning decisions

● Comparison data is favorable but does not indicate we have reached our goal

NSSE Benchmark Comparisons

58%42%69%42%34%86%86%73%69%21%0%50%100%Academic ChallengeActive / CollaborativeStudent-FacultyInteractionsEducational ExperiencesCampus EnvironmentFirst-Year StudentsSenior Students

NSSE Benchmark

MORN Comparisons2006 Selected Peers Carnegie Peers NSSE

Score ScoreEffect size Score

Effect size Score

Effect size

Level of Academic Challenge

First-Year 54.0 53.9 53.6 51.8 Senior 56.5 56.2 57.7 55.8

Active and Collaborative Learning

First-Year 45.7 44.2 44.6 41.3 0.27Senior 56.3 53.6 54.2 50.4 0.35

Student-Faculty Interaction


Enriching Educational Experiences

First-Year 26.2 28.2 28.4 26.7 Senior 46.7 45.9 44.1 39.9 0.39

Supportive Campus Environment


Effect size is a measure of the strength of the relation


Student Learning

Student Learning Outcomes

●Public statements of what we expect our students to have learned by the time they graduate from Morningside College

●Provide the focus for curricular structures and requirements

Assessment 101, continued

Identify goals /

purpose

Determine student learning

outcomes/ questions

Determine assessment

methods

Identify curricular and/or co-curricular learning activities

Implement learning activities

Gather evidence

Analyze and interpret evidence

Identify strengths and areas for improvement

Identify expected level of

performanceQuestion

Assumptions

Student Learning Outcomes

Eight Morningside Student Learning Outcomes

Across the Curriculum

Competencies

Distribution and Flag Student

Learning Outcomes

Academic Program

Outcomes

The Eight Morningside Student Learning Outcomes

Morningside graduates:1. Demonstrate analytic, synthetic, creative, evaluative, and

quantitative thinking;2. Communicate effectively;3. Behave ethically and responsibly;4. Use knowledge of cultures to enhance their understanding of

themselves and others;5. Apply knowledge and skills from multiple, diverse disciplines and

practical experiences to understand complex issues and to solve problems;

6. Exhibit a passion for life-long learning;7. Effect positive change through leadership or active participation in

their communities; and8. Articulate their own spirituality and personal values, while

understanding those of others.

Relationship between the Morningside Eight SLOs and Graduation Requirements

Graduation Requirements

Morningside Eight SLO

1 2 3 4 5 6 7 8

FYE: Passport 1 1 1 1

FYE: Composition & Communication 1

Distribution: Quant. Reasoning 1

Distribution: Ethics & Values 1 1

Distribution: Global Awareness 1 1 1 1

Distribution: American Experience 1 1 1 1

Distribution: Empirical Reasoning 1 1

Distribution: Creative Expression 1 1 1

Relationship between the Morningside Eight SLOs and Graduation Requirements

Graduation Requirements

Morningside Eight SLO

1 2 3 4 5 6 7 8

Writing Across the Curriculum 1 1

Technology Across the Curriculum 1 1 1

Info. Literacy Across the Curriculum 1 1 1

Flag: Service Learning 1 1 1 1

Flag: Religious Traditions 1 1 1

Academic & Cultural Arts Series 1 1

Major 1 1 1 1

Major Capstone Course 1 1 1 1

Cluster 1

May Term 1

Status of Eight Morningside Student Learning Outcomes

Assessment● Combination of direct and indirect methods

to gather information from multiple sources● The Morningside Experience Study is a

longitudinal study of a cohort of entering students● Interviews and surveys● Electronic portfolio built on the Eight Outcomes

● Morningside will be participating in CIC/CLA Consortium 2008-2011● Critical thinking (part of Outcome 1)● Analytical thinking (part of Outcome 1)● Written communication (part of Outcome 2)

Centralized Indirect Measures

Survey When

Student Character.

& Traits

Learning Environ. &

Exper. SLOs

ACT Class Profile Report Annual X

CIRP Survey Annual

Alumni Placement Survey Annual

Alumni Survey Annual

College Student Survey (CSS) 2007-08

NSSE 2008-09

Graduating Senior Survey/Interview 2009-10

Status of Eight Morningside Student Learning Outcomes

Assessment● What is going well

● Robust indirect assessment system in place● IDEA added another component that provides

outcomes feedback to instructors each term● Excellent retention of students in longitudinal

cohort study

● What is going less well● Did not reach target of having 30 students in the

longitudinal cohort study ● Delay in receiving portfolio data (new curriculum)

may be frustrating faculty. Propose to CPC that a group of seniors be recruited for next year to complete portfolios

Status of Across the Curriculum Competency

Assessment● Requirements that students must meet

department competency standards prior to graduation

● Writing Across the Curriculum (WAC) program over 20 years old

● Curriculum revision adopted by faculty in 2003 added Technology Across the Curriculum and Information Literacy Across the Curriculum

● Despite repeated requests, beginning in Spring 2006, most departments did not submit plans for assessing TAC and ILAC until Spring 2008.


Assessment● What is going well

● Model is competency based at the exit-level and discipline-specific

● The approach is developmental and, for most departments, embedded within courses

● Some degree program assessment plans have cited WAC, TAC and/or ILAC methods. In those programs, data from one or more of those methods may be used to assess either the Eight Morningside Student Learning Outcomes all departments are responsible for assessing (1, 2, 3, and 6) and/or for assessing discipline specific student learning outcomes.


Assessment● What is going less well

● Data from the TAC and ILAC exit-level assessments will not be available until the first wave of entering students in Fall 2008 reach the appropriate stage according to their department.

● It is not clear whether the WAC model of initial screening, remedial coursework and common freshman coursework also applies to TAC and ILAC. This could be a component of the proposed review of the first-year seminar.

● Despite a number of recent discussions about writing, no meta-assessment of student WAC submissions has been conducted or is planned at this time. Last review of WAC was conducted in January 2002 and does not appear to have examined effectiveness of the placement process, remedial coursework, or conduct a meta-assessment of a random sample of exit-level WAC assignments.

Status of Distribution and Flag Student Learning Outcomes

Assessment● General studies requirements that all

students must fulfill● Student learning outcomes have been

identified for each distribution or flag area● Each course approved to meet a

requirement also has an approved assessment plan

● Each semester, instructors are required to report their assessment results to CPC

● CPC reviews results on a cyclical basis


Assessment●What is going well

●Instructor-led, course-embedded assessment process allows results to more directly inform changes at the level learning occurs

●Reports indicate instructors are engaged in critical reflection of their assessment results

●Results have indicated need for improvements in curriculum, pedagogy, student learning outcomes and/or assessment methods

●Was key to successful pass of HLC 3-yr progress report


Assessment●What is going less well

●Compliance with reporting requirement has been poor (less than half)

●CPC has struggled with review effort (reduced membership, inexperience with process, etc.) but has proposed a possible solution that involves using groups of instructors to perform initial review

Status of Academic Program Assessment

● An academic program is an approved major and degree combination

● Required to have student learning outcomes, a curriculum map, Across the Curriculum assessments, and a capstone experience which includes assessment of Morningside Student Learning Outcomes 1, 2, 3, & 6.

● Academic programs with discipline specific accreditation must meet their accreditor’s requirements.

● Larger academic programs (three-year average of ten or more graduates) must maintain assessment plans and report student learning results

● Assessment activities and results level are reviewed as part of Morningside’s departmental review process.

Status of Academic Program Assessment

● What is going well● Programs with specialized accreditation are

adhering to assessment requirements● Program review has included review of

assessment plans, procedures and results

● What is going less well● Programs that have not submitted basic

information: ART, BUSN, CCOM, MATH, PHYS, SPAN, & RELI

● Programs that have not submitted annual results reports: BIOL


Internal Factors




Student Learning




● Annual performance reviews for faculty members have undergone extensive changes● Performance rubrics established● Piloting of a new student rating of instruction

system (IDEA) used in all courses● Piloted an advising assessment instrument● Faculty discussion on scholarship expectations

● Annual performance reviews for staff and administrators revised to include a merit-based system for determining raises.

Status of Personnel Performance

● What is going well● Faculty performance rubrics continue to mature● IDEA pilot continues

● What is going less well● It is disappointing that Faculty Senate decided to

discontinue use of the advising assessment instrument (rationale was low student response rates)

● The institution should consider conducting a systematic review of the reward system to determine the extent to which it is properly aligned with the institution’s mission, vision and goals

2007-2008Institutional Goals

4. Assessment of student learning must become a priority for this institution.

an update on morningside’s institutional effectiveness program daniel petra, edd office of...

Documents

effectiveness slide

morningside assessment

assessment results

assessment rests

term assessment

assessment definitions

purpose of assessment

effectiveness measurement