MEASURING WHAT MATTERSMapping the New World of Higher Educ. Assessment and Outcomes

Kenneth C. GreenTHE CAMPUS COMPUTING PROJECT

THE CAMPUS

COMPUTING PROJECT

Key Questions

• What is quality in higher education?

• How do we define it? Measure it?

• How do we explain it to ourselves? Others?

• How do we improve the quality of academic programs and institutional outcomes?

THE CAMPUS

COMPUTING PROJECT

Complaints About Quality

• FACULTY: Today’s college students are not well-prepared for college-level work.

• EMPLOYERS: College graduates come to us lacking key skills.

• EXECUTIVES: US college graduates are not prepared to compete in the global economy.

• GOV’T OFFICIALS: Universities expect money without accountability.

THE CAMPUS

COMPUTING PROJECT

Plus ca Change!

THESE ARE NOT NEW COMPLAINTS:

• Endemic to higher education

• Recurring theme in commission reports over the past 40 years

WHY NOW?• GW Bush administration has made educational

assessment and accountability a priority

• Accelerating concern about higher educ. assessment and outcomes in the states

THE CAMPUS

COMPUTING PROJECT

CONTEXT: Bring Data!

THE CAMPUS

COMPUTING PROJECT

“Back in Texas we like to say ‘In God we trust; all others bring data.’ ” Margaret Spellings, Sec. of Education

CONTEXT: Bring Data!

“Back in Texas we like to say ‘In God we trust; all others bring data.’ ” Margaret Spellings, Sec. of Education

TRUE SOURCE: “In God we trust; all others bring data.” W. Edwards Deming

The Spellings Commission

• GOOD NEWS: not so dire

• BAD NEWS: we’ve heard these critiques before

• REAL NEWS: pay attention, as this report has “legs”

THE CAMPUS

COMPUTING PROJECT

Spellings: Key Messages

• ACCESS: “too few Americans prepare for, participate in, and complete higher education”

• AFFORDABILITY: “costs have outpaced inflation for two decades…. Our higher education financing system is increasingly dysfunctional”

• ACCOUNTABILITY: “there is inadequate transparency and accountability for measuring institutional performance”

THE CAMPUS

COMPUTING PROJECT

The Quest for Accountability

We are concerned about the widespread resistance to cost

effectiveness thinking in higher education because it is so

profoundly anti-intellectual. It rejects reason, and it puts a low

value on the time of faculty trained to reason well…

We must guard against the widespread tendency to trivialize

the problem of efficiency in higher education. It is not only a

financial problem but an intellectual one. The questions about

efficiency lead to a host of questions about teaching and learning,

and the ultimate questions about the nature and purpose of higher

education. These are too important to the colleges and

universities – and too intellectually challenging – to be dismissed

as illegitimate.President’s Task Force on Higher Education, 1971

THE CAMPUS

COMPUTING PROJECT

Outcomes from OnLine learning compared to Face-to-Face courses

0%

20%

40%

60%

80%

100%

All Institutions Public Private For-Profit

Superior Same Inferior

11.0%

50.6%

38.5%24.5% 56.2%

17.3%

35.5% 78.3%

62.0%

13.4% 8.3% 4.3%

Source: The Sloan Consortium (2004) www.sloan-c.org

THE CAMPUS

COMPUTING PROJECT

PROVOST PERSPECTIVES

HOW DO THEY

KNOW?• Are provosts in classes and chatrooms?

• Are they looking at data?

• Are they doing formal assessments and evaluations?

• Are they talking with students, faculty & employers?

How Do We Measure Program Quality and Institutional Outcomes?

Resources

• Student characteristics

• Faculty profile

• Financial resources

• Physical plant

Performance

• Retention and graduation rates

• Retention in key fields or among key groups/populations

• Faculty research and publications

• Testing

THE CAMPUS

COMPUTING PROJECT

Assessment Strategies

• SHADOW ISSUE: No Child Left Behind (NCLB) and the rise of “high stakes testing”

• DATA MINING: exploiting the rich array of student and institutional data already available

• TESTING: turn to third party instruments for independent measures and metrics

THE CAMPUS

COMPUTING PROJECT

HYBRID: Multiple Data Sources

• ACKNOWLEDGE: multiple outcomes, both individual and institutional

• EXPLOIT: existing resources & data

• INVEST: in resources and strategies that address real institutional needs

• RECOGNIZE: benign neglect is not an effective institutional response

The ERP Problem

THE CAMPUS

COMPUTING PROJECT

The ERP Silo

FinanceStudentInfo. Sys.

Develop-ment

HR

Adm Info Systems

THE CAMPUS

COMPUTING PROJECT

The ERP Silo Turtle

FinanceStudentInfo. Sys.

Develop-ment

HR

Adm Info Systems

Portals ePortfolio

CourseMgmt

eServices

ContentAlumniServices

Source: Green, 2004

Icons of the New Internet Economy

THE CAMPUS

COMPUTING PROJECT

The consumer and corporate experience drives expectations about campus resources and services, including analytics.

Tracking transactions to provide timely data about shoppers for managers & shareholders

Extracting value from transaction data toprovide feedback to clients

QuickTime™ and aTIFF (LZW) decompressor

are needed to see this picture.

Investing in research to provide feedback, improve programs and enhance client services

The Spellings Commission

Message to Campus Leaders

• LEVERAGE IT: IT moves is now part of the conversation about data, assessment, and outcomes

• KEY TOOLS: data warehousing/mining & ERP analytics

• RIP: The IR office as we know it

THE CAMPUS

COMPUTING PROJECT

Outcomes and ERP Analytical Tools

A key component of the outcomes and assessment

solution [mandated by the Spellings Commission]

resides in the emerging analytical IT tools

increasingly deployed in the corporate sector and

now coming to higher education. These tools can,

do, and should expand the mission of information

technology in colleges and universities to include

assessment.Bring Data: A New Role for IT After the Spellings Commission Kenneth C. Green EDUCAUSE Review, Nov-Dec 2006

THE CAMPUS

COMPUTING PROJECT

Leadership

THE CAMPUS

COMPUTING PROJECT

The issue before us in the wake of the Spellings

Commission report concerns when college and

university IT leaders will assume an active role, a

leadership role in these conversations [about

assessment and outcomes], bringing their IT resources

and expertise - bringing data, information, and insight -

to the critical planning and policy discussions about

institutional assessment and outcomes that affect all

sectors of US higher education.Bring Data: A New Role for IT After the Spellings Commission Kenneth C. Green EDUCAUSE Review, Nov-Dec 2006

www.campuscomputing.net

THE CAMPUS

COMPUTING PROJECT

Thank You For Your Participation

Please complete our online evaluation