gauging effectiveness of instructional grants

Gauging the Effectivenessof Instructional and

Institutional Development Grants

Lynda Milne

Minnesota State Colleges & Universities

Center for Teaching & Learning

POD Network Conference, Pittsburgh

October 26, 2007

Our center, like many, has a long history of awarding grants to faculty. In this session we will discuss two grant programs and the systems we developed to categorize and analyze data about the purposes, topic areas, activities, and outcomes of grant projects.

Acknowledgements

David Laverny-RafterCTL Faculty Project Specialist

Professor, Political ScienceMinnesota State University, Mankato

Thomas WortmanAssistant Director for Grant Programs

Minnesota State Colleges & UniversitiesCenter for Teaching and Learning

Why Instructional

Grants? Tradition (scholarship,

research) Foster innovation Demonstration projects/

experiments/pilots Allocate scarce resources Competition fosters

excellence …(certainly many other)

How Little We Know

Levinson-Rose & Menges, 1981– Frequency, popularity of

programs

Eble & McKeachie, 1985– Positive faculty perception

Jacobsen, 1989– Little support from faculty

Weimer & Lenze, 1991– Grants at 40% to 78%

institutions– 70% to 90% rated highly

McAlpine & Gandell, 2003– Grants as SoTL; potential

impact

How Much We Do

What teaching incentive grants do you manage?

What kinds of assessment requirements are included in your guidelines?

How do your grantees report to you?

How do you report to your campus/provost/others the benefits and outcomes of grants?

Are grants having a positive impact?

How Much We Do

What teaching incentive grants do you manage?

What kinds of assessment requirements are included in your guidelines?

How do your grantees report to you?

How do you report to your campus/provost/others the benefits and outcomes of grants?

Are grants having a positive impact?

CTL Grantmaking History since

2000 Major Bush funds initiatives focused on active learning, 1999-2005– $2M for 200+ grants

Systemwide teaching grants– $350K for 56 projects

Systemwide course redesign– $250K for 11 grants

Multiple outcomes Multi-focused summative

evaluations http://www.ctl.mnscu

.edu/programs/grants/

Minnesota State Colleges & Universities

System Merged public higher

education system (all except University of Minnesota), 1995

7 state universities 25 community/technical

colleges 240,000 students 8,000+ faculty Graduate majority of state’s

teachers, nurses, “first responders” and law enforcement professionals

2004 Legislature Targets Funds

$6M per year for “competitive compensation for initiatives to promote excellence in student learning”

2005-2007 two new programs– College Faculty Awards for

Excellence (individual)– Initiatives to Promote

Excellence in Student Learning (institutional)

Guidelines

Efforts above and beyond regular work responsibilities

Focus on student learning Alignment with strategic

goals Work plan themes/critical

targets– Reading and writing

(transitions—first year and entry into major)

– STEM outcomes– Critical thinking

Strategic Directions

Increase access and opportunity

Promote and measure high-quality learning programs and services

Serve state and regional economic needs

Innovate for current and future educational needs and efficiency

Work Plan Eliminate the achievement

gap Provide universal access from

pre-kindergarten through the first two years of college

Promote increased participation in science, technology, engineering and math, known as STEM, fields

Expand corporate learning Develop colleges and

universities for Minnesota's future

Build organizational capacity for change and innovation

2005 Faculty Development

Survey Critical thinking topic of

highest interest

Survey says…

Course Outcomes: 2004-

05 DFW Rates First-semester college writing

courses: 10% – 30% Developmental reading and

writing courses: 25%– 54% College algebra: 15% - 58% Developmental math: 15% - 64% Physical sciences: 12% – 48%

Source: Analysis of large-course (5+ sections) grade distributions, Fall 2004 – Spring 2005. Ranges across 32 institutions.

Course Outcomes:2004-05 DFW

RatesCourses with 5 or More

Sections# Students Enrolled All Sections

2-YearColleges

4-Year Univs

English and Speech

91,162 25.6% 12.1%

Mathematics

46,346 40.1 32.9

Biological/Life Sciences

18,244 27.3 19.4

Physical Sciences

11,935 20.9 20.2

Structuring Guidelines

Review the excerpts of the guidelines for each of the two programs.

Then on the “How Would You Assess These Grants?” handout, answer the questions that we faced about evaluating and reporting on these grants.

Reporting Aligned with

Purposes Outcomes Principles Strategies Dissemination Sustainability

Analyzing the Reports

Description– How many, what

proportion of grant managers

Improved learning Created new curriculum Innovated in strategic areas

Analysis– Trends– Relationships

Reporting– Impact

Outcome-Based Evaluation

Description– Inputs– Outputs– Outcomes

Analysis– Trends– Relationships

Reporting– Impact

Influences shaping our

evaluation work Scholarship Assessed NCAT Logic Model/Outcome-

Based Evaluation methods

Scholarship Assessed

Scholarship should be the outcome we measure

Pre-grant (formative assessment)– Clear goals– Adequate preparation– Appropriate methods

Post-grant (summative– Significant results– Effective presentation– Reflective critique

- Glassick, Huber, Maeroff (1997). Scholarship Assessed

NCAT Course Redesign

NCAT ‘s Program in Course Redesign showed clearly 3 big outcomes from a major grant project:

1. Improvements in learning– Higher grades– Better test scores– Assessments of learning

quality/ depth

2. Lower DIW rates

3. Lower instructional costs

Outcomes-Based Evaluation Methods

Inputs (resources) Outputs (activities, products) Outcomes (results of the

outputs, changes that occur--in learning, attitudes, behavior, etc.)

Impact (long-term benefits of the program)