Degrees of Quality

Terrel L. Rhodes, Association of American Colleges and Universities

October 19, 2012

1. Have you heard of

“Swirling”?

“The American Dream is imperiled.”

“To be blunt: The colleges must

redesign their institutions,

their mission and their students’

educational experiences

to ensure that they meet the needs

of a changing society.”

http://www.nola.com/newsflash/index.ssf/story/performance-funding-making-its-way-into-higher/150ea4cc13ebc91cae1f6d386a6ca090

The message to higher education leaders is simple:

"If you want more money, prove you deserve it."

"They're frustrated about completions.

They're frustrated about transfers.

They're frustrated that students

are taking six or seven years to graduate!"

The Evidence is Compelling...

Too Many Students Are

Underachieving!

The Evidence is Compelling...

Too Few Institutions Are Responding!

2. Have you heard of

“Badges”??

What Do Employers Want?

20%

20%

59%

Which is more important for recent college graduates who want to pursue advancement and long-term career success at your company?

BOTH in-depth AND broad range of skills and knowledge

Broad range of skills and knowledge that apply to a range of fields or positions

In-depth knowledge and skills that apply to a specific field or position

“Irrespective of college major or institutional selectivity, what matters to career success is students’ development of a broad set of cross-cutting capacities…”

—Anthony Carnevale, Georgetown University

Center on Education and the Workforce

9

Expecting students to complete a significant project before graduation that demonstrates their depth of knowledge in their major AND their acquisition of analytical, problem-solving, and communication skills (62% help a lot)

Expecting students to complete an internship or community-based field project to connect classroom learning with real-world experiences (66%)

Ensuring that students develop the skills to research questions in their field and develop evidence-based analyses (57%)

Expecting students to work through ethical issues and debates to form their own judgments about the issues at stake (48%)

Source: Raising the Bar (AAC&U, 2010)

Employers Assess the Potential Value of Emerging Educational

Practices% saying each would help a lot/fair amount to prepare college students for success

84%

81%

81%

73%

2. Have you heard of the “Completion

Agenda”?

3. Have you heard of “the

DQP”?

What is the Degree

Qualification’s Profile (DQP)?

How is the DQP

different?

CompetencyThat is neither trivial or

snackable...

LEAP Promotes Essential Learning OutcomesA Guiding Vision and National Benchmarks for College Learning and Liberal Education in the 21st Century

High Impact PracticesHelping Students Achieve the Essential Learning Outcomes

Authentic AssessmentsProbing Whether Students Can APPLY Their Learning – to Complex Problems and Real-World Challenges

Inclusive ExcellenceDiversity, equity, quality of learning for all groups of students

High Impact Practices

First-Year Seminars and Experiences

Common Intellectual Experiences

Learning Communities

Writing-Intensive Courses

Collaborative Assignments and Projects

High Impact PracticesUndergraduate Research

Diversity/Global Learning

Service Learning, Community-Based Learning

Internships

Capstone Courses and Projects

ePortfolios

LEAP Principles of Excellence

Principle One Aim High—and Make Excellence

Inclusive

Principle Two Give Students a Compass

Principle Three Teach the Arts of Inquiry

and Innovation

Principle Four Engage the Big Questions

Principle Five Connect Knowledge with

Choices and Action

Principle Six Foster Civic, Intercultural,

and Ethical Learning

Principle Seven Assess Students’ Ability to

Apply Learning to

Complex Problems

Why Did AAC&U Join the DQP Effort?

Focus on Student Performance

Research ProjectsPapers

PerformancesCreative

The DQP Emphasizes....

Integration of learning

Application of learning

The DQPhttp://www.luminafoundation.org/publications/The_Degree_Qualifications_Profile.pdf

1.Specialized Knowledge2.Broad Integrative Knowledge3.Intellectual Skills and

Abilities4.Applied Learning5.Civic Learning

The Lumina Degree Profile – in Brief –

Outlines Competencies Required

for the Award of Degrees

The Need?No “consistent public

understanding” of what degrees mean....

High Standards do not need standardization!

Associates Level

Bachelors Level

Masters Level

An Example...Communication Fluency

Associate Level: The student presents substantially error-free prose in both argumentative and narrative forms to general and specialized audiences

Bachelor’s Level: The student constructs sustained, coherent arguments and/or narratives and/or explications of technical issues and processes, in two media, to general and specialized audiences

Master’s Level: The student creates sustained, coherent arguments or explanations and reflections on his or her work or that of collaborators (if applicable) in two or more media or languages, to both general and specialized audiences

The DQP Strategy

Moving Students’ Actual Work – and

Faculty Judgment – about student progress – to the Center of Attention

Assessments Worthy of Our Mission

The Five Areas are Interrelated, Not Separate

For example: Knowledge and Intellectual skills

are integrated in the context of application

– e.g. research, field-based assignments, projects, and civic problem-solving

Current Realities and Framework Ask Us to Shift from My Work – Each Course is a Silo – to OUR

Work – Intentional Practices that Both Develop and Demonstrate

Students’ Competence

The Work Ahead?

The higher ed market is reinventing what a university is, what a course is, what a

student is, what value is. I don’t know why anyone would think that the online

experience is about reproducing the classroom experience.

Richard A. DeMilloDirector, Center for the 21st Century University

Georgia Institute of Technology

Goal:

Foster graduates who consider

ethics, legal, and socio-cultural issues and use

critical thinking in decision-making.

Measure :

Grades that focus on presentation style,

grammar, and citation form and

number.

We have had our why's, how's, and what's upside-down,

focusing too much on what should be learned, than on how, and

often forgetting the why altogether.

In a world of nearly infinite information, we must firstaddress why, facilitate how, and let the what

generatenaturally from there.

Michael Wesch, “From Knowledgeable to Knowledge-able,” Academic Commons, January 2009 (academiccommons.org)

38

4. Have you heard of

ePortfolios?

ePortfolio Focus

What We Know and Have Evidence to Believe

•Used on over half of American higher education institutions•Improve retention and graduation•Deepen student learning – retain and integrate information, apply information beyond single course

Institutional Usage

Institutional Size

• Fostering reflection• Student responsibility• Tracking growth• Showcasing learning• Self-assessment• Integrative learning

What is most important to individual respondents?

Motivation for ePortfolio

How e-portfolio ideas and practices have improved or could improve

student learning

“Could learning e-portfolios encourage students to have deeper, more reflective learning, and stimulate more significant connections across learning experiences?” – University of Oregon

We found specific changes in learning and pedagogy

occur as a result of participation in

e-portfolios

Essential Portfolio Practices

• Purposeful collection• Multiple measures to track development and

improvement• Self-assessment and reflection to foster analysis,

synthesis, evaluation, etc.• Integrative opportunities/requirements• Build evidence of an empowered, informed,

responsible learner • Can be used a course, program, institutional level

assessment

Positioning Integration in the Student’s Experience

One of the great challenges in higher education is to foster students’ abilities to integrate their learning

across contexts and over time. Learning that helps develop integrative capacities is important because it

builds habits of mind that prepare students to make informed judgments in the conduct of personal,

professional, and civic life; such learning is, we believe, at the very heart of liberal education.

~Mary Taylor Huber and Pat Hutchings

Integrative Learning: Mapping the Terrain (2005)

5. Have you heard of the

VALUE Rubrics?

Rubrics Basics

Criteria

VALUE Rubrics & AssessmentVALUE Rubrics & Assessment

Rubrics Basics

Levels

VALUE Rubrics & AssessmentVALUE Rubrics & Assessment

Rubrics Basics

Performance Descriptors

VALUE Rubrics & AssessmentVALUE Rubrics & Assessment

Reliability Study• 40 Faculty• 4 Traditional Disciplinary Divisions –

Humanities, Social Sciences, STEM, Professions• Three VALUE rubrics – Critical Thinking, Civic

Engagement, Integrative Learning• Common set of student portfolio work• Agreement = .66 without norming; .8 normed• Another set of 5 campuses, using same set of

rubrics with 500 samples of student work – still analyzing

AAC&U Perspectives

• Standardized vs. Standards-based Assessments

• Faculty Developed• Focused on Competence vs. Deficits• Based on Student Work• Demonstrated over time vs. Snapshot• VALUE Rubrics

Building the Evidentiary Base• University of Kansas – Representing Results

Per

cen

t o

f R

atin

gs

Critical Thinking: Issues, Analysis, and Conclusions

Inter-rater reliability = >.8

Building the Evidentiary Base• University of Kansas – Representing Results

Per

cen

t o

f R

atin

gs

Critical Thinking: Evaluation of Sources and Evidence

Building the Evidentiary Base

Perc

en

t of

Rati

ng

s

“VALUE added” for 4 years - writing

Building the Evidentiary Base

• University of Kansas – • “analysis of the data from the AACU VALUE rubrics

affirmed that a team approach to course design can yield larger improvement in some forms of student writing and thinking”

• “We also saw that the rubrics work best when there is close alignment between the nature of the assignment and the dimensions of intellectual skill described in the rubric”

• “Finally, at a practical level we are very encouraged that this process is manageable and sustainable”

A Learning College: Adjusting

for Student Growth &

Development

Faculty Read & Assess Student Work• Faculty provide

appropriate developmental guidance and a final assessment of the work – student learning, pedagogical improvement

Programs Read & Assess Student Work• Programs can make

changes based on what they find & with resources if possible The College Reads

& Assesses Student Work• The college is able to

assess progress in core competencies across the curriculum, not just in a program - accountability.

Faculty

Reviews Assignment & Guides Student

Progress

Adapts Assignment & Makes Changes

Where Appropriate

Program

Program X discovers that oral

communication is weak in an intro

course.

Works with faculty from Oral

Communication to design an

intervention. Intensive module design

supported with small grant.

College

Benchmark Assessment Readings reveal gains in Critical Literacy and Research & Information Literacy but also reveal issues

with the rubrics

Rubrics will be redesigned to better support the kinds of work faculty want to

assign. Additional work in CTL seminars

and in the college around the core competencies.

The Feedback Loop

Assessment & the Learning Portfolio

Two different facets of the ePortfolio

ePortfolios can be used for accountability (summative - supported by student work placed in the ePortfolio

evaluated through rubrics)

ePortfolios are also used for student learning, growth, and development (formative and on-going)

Faculty Collaborati

on

We are Being Asked to Shift...

from My Work

to OUR Work

Assessment Practices

That Verify Achievement