www.inquiry2improvement.com dr. rob johnstone aspen / atd leadership symposium wye river, md april...

www.inquiry2improvement.com

Dr. Rob JohnstoneAspen / ATD Leadership Symposium

Wye River, MD April 1, 2014

From a Culture of Evidence to a Culture of Action & Inquiry

APPENDIX: Examples of Outcomes Data

That Matter

www.inquiry2improvement.comNati onal Center for Inquiry & Improvement

Acknowledgements The content in this presentation has been collected from a

variety of sources, all of which are publicly available on websites or have been presented in public settings.

These sources include local college IR efforts, state system efforts, and national projects such as: Completion by Design The Aspen Prize for Community College Excellence The Carnegie Foundation’s Statway / Quantway Collegemeasures.org

Content has also been developed by and with a host of national partners, including CCRC, the Aspen Institute, the RP Group (CA), and AIR (American Institutes for Research)

http://www.inquiry2improvement.com/

3www.inquiry2improvement.com

Examples of Data that Matter

April 2014

www.inquiry2improvement.comNati onal Center for Inquiry & Improvement 4

Aspen’s Take on Data that Matters:

Progress / Completion Outcomes Data

Labor Market Outcomes Data

Learning Outcomes Data

Equitable Outcomes Data



Progress / Completion Data: A “Quick” Look

April 2014


Completion Example #1CUNY ASAP Program

Graduation Rates


ASAP Program Results

7http://www.cuny.edu/academics/programs/notable/asap/about.html


Completion Example #2Completion Rates in

Completion by Design


Example from CBD: Outcomes for LAS Concentrators

Outcomes by CTE Concentrators


Completion Example #3Valencia College


Outcomes of Prize finalist colleges: Exceptional improvements over time in completion


Progress Example 1:Grades in Sequenced Courses



Grades in Sequenced Courses Backdrop

• Course sequences analyzed in Mathematics, English, Biology, Chemistry, Physics & Economics

• Relationship between grade in 1st course in sequence and success in 2nd course in sequence was examined.

• Enrollment data tracked over five-year period• Selected highlight slides follow; full PPT

provided on request



Data Too Often Presented Like This…

Status in Math 310 Count% of Total

NSuccess Rate in Math 310

A in Math 308 5,659 10% 81%

B in Math 308 7,726 13% 60%

C in Math 308 7,234 12% 37%

Unclear Status in Math 308* 1,675 3% 16%

Math 310 Repeater / Non Success 14,568 25% 41%

Math 310 Attempt was First Math Course

16,812 29% 57%

Taking Math 308 / 310 in Same Semester

4,505 8% 68%

Other 487 1% 44%

Total 58,666 100% 53%

15


A

B

C

The grade received by students in Math 308

Pre-Algebra Introductory Algebra

Intermediate Algebra

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Success Rate of those same students in Math 310

37%

60%

81%

Math 306 Math 308 Math 310

Intermediate AlgebraMath 310

B

A

Transfer-Level

Those placing directly into

Math 31057%

Math 1300s

And Not Often Enough Like This…

A

B

C

The grade received by students in Math 2413

Calculus I Calculus II

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Success Rate of those same students in Math 2414

50%

78%

90%

Calculus Course Sequence

MATH 2413 MATH 2414

Calculus II

MATH 2414

B

A Those placing directly into Math

2414

72%

A

B

C

The grade received by students in English 307

One Level below

Transfer

One Level below

Transfer

Two Levels below

Transfer

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Success Rate of those same students in English 1301

46%

61%

74%

Developmental Education Writing Course Sequence

Those placing directly into English 1301

ENGL 307ENGL 306

Transfer Level English

ENGL 1301

B

A71%

A

B

C

The grade received by students in Biology 2401

Anatomy Physiology

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Success Rate of those same students in Biology 2402

50%

72%

88%

College Biology Course Sequence

BIO 2401 BIO 2402

Physiology

BIO 2402

B

AThose placing directly into Biology 2402

67%

A

B

C

The grade received by students in Chemistry 1411

General Chemistry I

General Chemistry II

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Success Rate of those same students in Chemistry 1412

46%

70%

86%

College Chemistry Course Sequence

CHEM 1411 CHEM 1412

General Chemistry II

CHEM 1412

B

AThose placing directly into

Chemistry 141266%

A

B

C

The grade received by students in Economics 2301

Macro Economics

Micro Economics

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Success Rate of those same students in Economics 2302

74%81%

89%

Economics Course Sequence

ECON 2301 ECON 2302

Micro Economics

ECON 2302

B

A Those placing directly into

Economics 230271%


Progress Example 2:Developmental Education

Cohort Tracking



INTRODUCTORY ALGEBRA

INTERMEDIATE ALGEBRA

PRE-ALGEBRA TRANSFER LEVEL

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

100% Attempt

the Course

43% Attempt

the Course 18%

Attempt

72% Pass the Course

27% Pass the Course 11%

Pass 5% Att4% Pass

Of the original cohort

Of the original cohort Of the

original cohort

Notes: Enrollments from Fall 2009 thru Spring 2012; Success is defined as A/B/C/CR grade.

Math cohort tracking starting in Pre-Algebra






0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

100% Attempt

the Course

50% Attempt

the Course 25%

Attempt

64% Pass the Course


Pass




Math cohort tracking starting in Introductory Algebra






0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

100% Attempt

the Course

50% Attempt

the Course

67% Pass the Course

37% Pass the Course



Math cohort tracking starting in Intermediate Algebra



DEV WRITING II COMPOSITION IDEV WRITING I COMPOSITION II

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

100% Attempt

the Course

53% Attempt

the Course

33% Attempt

71% Pass the Course


Pass the Course




English Writing cohort tracking starting at Developmental Writing I



DEV WRITING II COMPOSITION IDEV WRITING I COMPOSITION II

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

100% Attempt

the Course

65% Attempt

the Course

78% Pass the Course

54% Pass the Course



English Writing cohort tracking starting at Developmental Writing II



Progress Example 3:Look for Examples from

Four-Year Schools: Retention & Progress at

Georgia Statehttp://

www.edtrust.org/higher_ed_practice_guide

for more examples


http://www.edtrust.org/higher_ed_practice_guide





First Year to Second Year Retention, Georgia State University

Fall '00 Fall '01 Fall '02 Fall '03 Fall '04 Fall '05 Fall '06 Fall '07 Fall '08

80% 82% 81% 83%80% 81% 82% 83% 83%

Retention



First Year Retention & Progression Rates - Georgia State University

Fall '00 Fall '01 Fall '02 Fall '03 Fall '04 Fall '05 Fall '06 Fall '07 Fall '08

80% 82% 81% 83%80% 81% 82% 83% 83%

22%27% 28%

33% 35%39%

49%

62%67%

Retention Retained & Sophomore



Labor Market Outcomes Data: A Quick Look

April 2014


Labor Market Outcomes Example 1:

CollegeMeasures.org ESM

for Program-Level LMI Data



CollegeMeasures.org is one possibility to get LMI data

● Collaboration between the Lumina Foundation and AIR

● Six states currently signed on - AR, CO, FL, TN, TX, VA

● Any state can theoretically participate

● Check it out: http://www.collegemeasures.org/esm/

33


http://www.collegemeasures.org/esm/

http://www.collegemeasures.org/esm/


Wages of Individual Degree Programs at an Arkansas CC

34

Childcare/Support Services

Medical Office Management

Accounting

Business

Liberal Arts

Computer/IS

Criminal Justice

Nursing

Precision Production

$0 $10,000 $20,000 $30,000 $40,000 $50,000

$18,078

$20,517

$22,903

$24,374

$24,724

$27,952

$30,297

$44,988

$45,510

Earnings



Wage Earning Rate of Individual <1 Year Certificate Programs at a Colorado CC

35

Practical Nursing

Diagnostic/Intervention

Lab Science/Research

Architectural Engineering

Real Estate

Computer/IS

Medical Assisting

Criminal Justice

Health Aides

Business Admin.

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

16%

19%

20%

23%

24%

28%

29%

32%

37%

51%

Wage Earning Rate



Wages for AA Degrees in Business / Mgmt / Marketing in Tennessee, By College

36

College A

College B

College C

College D

College E

College F

College G

College H

College I

College J

College K

College L

College M

$0 $10,000 $20,000 $30,000 $40,000 $50,000

$27,518

$29,591

$30,487

$31,031

$31,897

$32,790

$33,044

$33,954

$35,984

$36,108

$36,848

$39,127

$39,462

Earnings



Wage Earning Rate of Licensed Vocational Nurses in Virginia, By College

37

College A

College B

College C

College D

College E

College F

College G

College H

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

33%

22%

42%

37%

46%

52%

54%

41%

44%

48%

56%

60%

62%

63%

63%

67%

Wage Earning Rate

Wage Rate Above Avg Wage Rate



Labor Market Outcomes Example 2:

Aspen Prize’s Approach for

Comparing Labor Market Outcomes



Aspen Prize Sample Model Comparing Labor Market Outcomes

Institution

Employment Metrics Earnings and Wage Metrics County-level Economic Context Metrics

Class of 2011 Composite

Employment Index

Class of 2006 Composite

Employment

Annualized Salaries and

Wages for 2011

Employed Graduates

Class of 2011

Relative Wages

Annualized Salaries and

Wages for Employed

2006 Graduates in

2011

Class of 2006 Relative Wages

County Civilian Workforce

Participation Rate

2011 County Unemployment

Rate

County 5 Year

Employment

Growth Rate

County Community College

239 209 $55,784 180% $79,204 161% 76% 8.4% 7.0%

Peer College 2 176 182 $24,969 85% $48,130 102% 80% 9.0% -8.1%

Peer College 3 175 154 $20,304 89% $29,475 89% 67% 7.5% 1.4%

Peer College 4 153 151 $21,637 77% $41,894 99% 76% 9.5% 9.9%

Peer College 5 218 208 $28,353 138% $36,830 100% 86% 4.2% 0.9%

Peer College 6 137 157 $27,533 101% $43,973 91% 77% 8.6% -3.0%

Peer College 7 175 170 $23,483 96% $38,429 89% 77% 7.6% -7.5%

Peer College 8 147 120 $22,919 68% $34,005 76% 46% 10.4% 1.6%

Peer College 9 164 181 $40,966 177% $58,470 151% 72% 7.5% 5.0%

Peer College 10184 146 $21,227 84% $35,818 86% 72% 8.3% -2.5%

Peer-group average

177 168 $28,717 109% $44,622 105% 73% 8.1% -1.4%

39


Lessons from the Aspen Prize for Community College Excellence

COLLEGE EXCELLENCE PROGRAM

Outcomes of Prize finalist colleges: Exceptional short-term labor market outcomes for graduates

Average salaries of recent graduates compared to the average for all new-hires in the region (TOP THREE PERFORMERS)

Lake Area Technical Institute (SD)

$20,540

$28,756

40% above avg.

Regional average

Walla Walla Community College (WA)

$23,211

$41,548

79% above avg.

Regional average

Brazosport College (TX)

$31,086

$56,576

82% above avg.

Regional average

40



Outcomes of Prize finalist colleges: Exceptional long-term labor market outcomes

Average salaries of graduates 5 years after graduation compared to the average for all workers in the region (TOP THREE PERFORMERS)

Walla Walla Community College (WA)

$36,803

$57,044

Miami-Dade College (FL)

$45,664

$63,016

Brazosport College (TX)

$46,832

$77,272

55% above avg.

38% above avg.

65% above avg.

Regional average

Regional average

Regional average

41


Learning Outcomes Data: A Quick Look

April 2014


Goals of Learning Outcomes Assessment

•An ideal Learning Outcomes Assessment process: Focuses on authenticity vs. compliance Is driven & supported by the practitioners

tasked with improvement Leads to the generation of insight that is

woven into improvement efforts Creates campus conversations about student

success and improving outcomes

43



Learning Outcomes Assessment Thoughts

• Lots of outcomes to assess: Institutional Learning Outcomes General Education Learning Outcomes Program Learning Outcomes Course-level Learning Outcomes

• Challenging & complex – but critical to work on generating insight and improvement at all levels

44




• Not many nationally normed LOA models aside from standardized tests (e.g. CLA & CAP) – many local models that are good

•Accreditation focus on LOA has resulted in a massive collection / reporting effort – taking away from the actual use of data for improvement…

45




• Check out how CCBC’s Executive Summaries at link tell the assessment & improvement story at the course level My favorite all-time story is Chem 108 – tells how an

attempt to improve outcomes actually made it worse, but then further redesign achieved very high levels – tells that LOA is a journey to improvement story

• CCBC also strong on LOA at GE levels through a rubric assessment model – see link for more info

46


https://ccbcmd.edu/loa/CrseAssess.html

https://ccbcmd.edu/loa/great.html


Learning Outcomes Examples

• Alverno College The gold standard example of LOA – check out the student,

community, and faculty responses in addition to the mechanics - link

• Valencia College detailed program level assessments, excellent sharing of LOA

outcomes with peers – link

• Santa Barbara City College closing the loop by focusing on improvement plans – link

• West Kentucky Community & Technical College overall approach to student learning; significant gains in

reading based on LOA assessment – link47


http://depts.alverno.edu/saal/index.html

http://valenciacollege.edu/academic-affairs/institutional-effectiveness-planning/institutional-assessment/loa/

http://slo.sbcc.edu/improvementPlans.html

http://www.aspeninstitute.org/policy-work/aspen-prize/2011-aspen-prize/2011-aspen-prize-finalists/west-kentucky-community


Equitable Outcomes: A Quick Look

April 2014


Equitable Outcomes Example 1:

Northern Virginia CC & Transfers

to George Mason University


Northern Virginia Community College Students Transferring into George Mason University 2005–2012

Source: SCHEV 2014



Statway / Quantway looks at

Program Effects by Gender & Race


Statway looks at Pathway thru Algebra Success by Gender &

Ethnicity

52

Note. Data from unknown gender status were not presented; very few were unknown. “Other” also contains data from unknown status.

Black Hispanic White Other*

7%14%

18% 18%

45% 47%

63%

52%

Non-Statway Statway

Female Male

13% 16%

55%49%

Non-Statway Statway Statway Non-Statway

52% 14%

Gender and Race/Ethnicity

53

Note. Data from unknown gender status were not presented; very few were unknown. “Other” also contains data from unknown status.

Bla

ck

His

pa

nic

Wh

ite

Oth

er*

Bla

ck

His

pa

nic

Wh

ite

Oth

er*

Female Male

7%12%

17% 20%

9%17% 20% 17%

49% 49%

67%

52%

38% 42%

59%52%

Non-Statway Statway

0% 50% 100%0%

50%

100%

1

235

67

8911

13

14

15

17

1819

Non-Statway

Sta

tway

College Level Math Success By College

54

Statway Non-Statway

52% 14%

No Difference

Triple the Success

55

College Level Math Success - Faculty

0% 50% 100%0%

50%

100%

Non-Statway

Sta

tway

Statway Non-Statway

52% 14%



Aspen Prize’s Approach for

Comparing Equitable Outcomes at

the College Level



An Aspen Prize Finalist Achieves Near Equity in Outcomes

57




Outcomes of Prize finalist colleges: Far greater equity in outcomes than the national average

Three-year completion and/or transfer rates for underrepresented minority students compared to the national average

58

National Average Finalist Average Top 3 Average

34%

44%

49%

• Brazosport College (TX)• Santa Barbara City College (CA)• Santa Fe College (FL)

-5%

0%

5%

10%

15%

20%

1%

6%

13%



Outcomes of Prize finalist colleges: Exceptional equity in access for underrepresented populations

% Difference between underrepresented minority enrollment in institution and percentage of the population in the college’s service area (TOP THREE PERFORMERS)

0% means the college’s enrollment has the same proportion of underrepresented minority students as the general population; i.e., minorities are fully represented at the college

59


Find Out More

60

• The National Center for Inquiry & Improvement websitewww.inquiry2improvement.com

• Dr. Rob Johnstone, Founder & [email protected]

• CBD Inquiry Guides on Applied Inquiry & Nuances of Completion: http://www.inquiry2improvement.com/publications-resources


http://www.inquiry2improvement.com/publications-resources

http://www.inquiry2improvement.com/publications-resources

www.inquiry2improvement.com dr. rob johnstone aspen / atd leadership symposium wye river, md april...

Documents

completion slide

inquiry improvement

national center

design slide

research slide

request slide

math course16

progress completion