improving student achievement with new approaches to data

John Whitmer, Ed.D.Academic Technology Services

California State University, Office of the Chancellor

WASC ARC ConferenceApril 11, 2013

Improving Student Achievement with New Approaches to Data:

Learning Analytics & the CSU Data Dashboard

slides @ slideshare.net/JohnWhitmer/


Outline

1. Context: California State University & Graduation Initiative

2. Chico State Learning Analytics Case Study

3. CSU Data Dashboard Project

4. Next Steps

5. Discussion


1. CONTEXT


California State University http://calstate.edu

23 campuses 437,000 FTE students 44,000 faculty and staff Largest, most diverse, &

one of the most affordable university systems in the country

Play a vital role in the growth & development of California's communities and economy

http://calstate.edu/


CSU Achievement Gap


By 2015, the CSU will improve graduation rates by 8 percentage points systemwide and halve the achievement gap. – Baseline 6-Year Graduation Rate: 46%– Target 6-Year Graduation Rate: 54%

– Baseline Achievement Gap: 11%– Target Achievement Gap: 5.5%

2


New Approaches to Using Data

Enable data-driven decision making for interventions earlier in the student experience by

1. Integrate new data sources & variables

2. Disseminate findings to a broader audience

3. Provide ability to interact with data analysis, conduct ad-hoc and custom reporting


2. CHICO STATE LEARNING ANALYTICS CASE STUDY


Economist. (2010, 11/4/2010). Augmented business: Smart systems will disrupt lots of industries, and perhaps the entire economy. The Economist.

200MB of data emissions annually!


Source: jisc_infonet @ Flickr.com

Source: jisc_infonet @ Flickr.com

Logged into course within 24 hours

Interacts frequently in discussion boards

Failed first exam

Hasn’t taken college-level math

No declared major


Case Study: Intro to Religious Studies• Undergraduate, introductory, high

demand

• Redesigned to hybrid delivery format through “academy eLearning program”

• Enrollment: 373 students (54% increase on largest section)

• Highest LMS (Vista) usage entire campus Fall 2010 (>250k hits)

• Bimodal outcomes compared to traditional course • 10% increase on final exam• 7% & 11% increase in DWF

• Why? Can’t tell with aggregated data

54 F’s


Learner Analytics

“ ... measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs.” (Siemens, 2011)


Pervasive Adoption of Learning Management Systems

Institution-Supported IT Resources and Tools. Reprinted from “The ECAR Study of Undergraduate Students and Information Technology,” Eden Dahlstrom, 2012 by the EDUCAUSE Center for Applied Research.


Guiding Questions

1. How is student LMS use related to academic achievement in a single course section?

2. How does that finding compare to the relationship of achievement with traditional student characteristic variables?

3. How are these relationships different for “at-risk” students (URM & Pell-eligible)?

4. What data sources, variables and methods are most useful to answer these questions?


LMS Use Variables1. Administrative Activities

(calendar, announcements)

2. Assessment Activities (quiz, homework, assignments, grade center)

3. Content Activities (web hits, PDF, content pages)

4. Engagement Activities (discussion, mail)

Student Char. Variables1. Enrollment Status

2. First in Family to Attend College

3. Gender

4. HS GPA

5. Major-College

6. Pell Eligible

7. URM and Pell-Eligibility Interaction

8. Under-Represented Minority

9. URM and Gender Interaction


Correlation: Student Char. w/Final Grade

Scatterplot of HS GPA vs. Course

Grade


Predict the trend

LMS use and final grade is _______ compared to student characteristics and final grade:

a) 50% smaller

b) 25% smaller

c) the same

d) 200% larger

e) 400% larger


Correlation LMS Use w/Final Grade

Scatterplot of Assessment Activity

Hits vs. Course Grade


Chart: LMS & Student Characteristics


Combined Variables Regression Final Grade by LMS Use & Student Characteristic Variables

LMS Use

Variables

25% (r2=0.25)

Explanation of change in final grade

Student Characteristic

Variables

+10%(r2=0.35)

Explanation of change in final grade

>


Predict the trend

LMS use and final grade is ______ for “at-risk”* students compared to not at-risk students?

a) 50% smaller

b) 20% smaller

c) No difference

d) 20% larger

e) 100% larger

Relationship indicates how strongly LMS use is correlated with final grade; lower value equals less impact

*at-risk = BOTH under-represented minority and Pell-eligible


Predict the trend

LMS use and final grade is ______ for “at-risk”* students compared to not at-risk students?

a) 50% smaller

b) 20% smaller

c) No difference

d) 20% larger

e) 100% larger

*at-risk = BOTH under-represented minority and Pell-eligible


Question 3 Results:Regression by “At Risk” Population Subsamples


At-Risk Students: “Over-Working Gap”

27


Activities by Pell and Gradegrade / pelleligible

A B+ C C-

Pell-Eligible Not Pell-Eligible Pell-Eligible Not Pell-Eligible Pell-Eligible Not Pell-Eligible Pell-Eligible Not Pell-Eligible

0K

5K

10K

15K

20K

25K

30K

35K

Value

Content

Content

Engage

Engage

Assess

Assess

Admin

Admin

Content

Content

Engage

Engage

Assess

Assess

Admin

Content

Content

Engage

Engage

Assess

Assess

Content

Content Engage

Engage

Assess

Assess

Admin

Admin

Measure Names

Admin

Assess

Engage

Content

Extra effort in content-related activities


Conclusions

1. LMS use is a better predictor of academic achievement than student characteristics.– LMS use frequency is a proxy for effort.

2. LMS data requires extensive filtering to be useful; student variables need pre-screening for missing data.

3. LMS effectiveness for at-risk students may be caused by non-technical barriers.

4. Small strength magnitude suggests that better methods could produce stronger results.


Next Generation Learning Analytics

Graphic Courtesy Sasha Dietrichson, X-Ray Research SRL


3. DATA DASHBOARD PROJECT


THE FRAMEWORK

Advancing by Degrees: A Framework for Increasing College Completion by Offenstein, Moore & Schulock

Institute for Higher Education Leadership and Policy and The Education Trust (http://bit.ly/10QtMXC)

http://bit.ly/10QtMXC

http://bit.ly/10QtMXC


This research describes academic patterns (or leading indicators) that occur early in the pipeline that can be tracked and monitored in real time against milestones on the graduation route.


Milestones Leading Indicators Year-to-year Retention Transition to college level coursework

(English and Math) Earn one year of college level credits Complete General Education Complete degree

Remediation Begin remedial coursework in the first term, if

needed. Complete needed remediation

Gateway Courses Complete college-level math and/or English in

the first or second year Complete a college-success course or other

first-year experience program

Credit Accumulation and Related Academic Behaviors Complete high percentage of courses

attempted (low rate of course dropping and/or failure)

Complete 20-30 credits in the first year Earn summer credits Enroll full time Enroll continuously, without stop-outs Register on-time for courses Maintain adequate academic progress


Driving Questions for Dashboard

1. What percentage of students reach each of the leading indicators?

2. What is the impact of reaching each of the leading indicators on success rate?

3. Does meeting any of the indicators reduce or eliminate gaps between student demographic groups?


PROOF OF CONCEPT


Purpose

Demonstrate potential value of combined reporting and statistics

Evaluate availability and integration of data

Pilot potential tools in real-world scenario

NOTE: production system may be dramatically different from POC, given lessons learned and scalability


1. Report Parameters


2. Retention Rates


3. Retention Rates by URM Status

slides @ slideshare.net/JohnWhitmer/4. Data Export Options


Concern: Male, 2nd Year Persistence


4. NEXT STEPS


What’s Now … And Next

Conducting 3 multi-campus pilots1. mCURL: Moodle Common Usage Reporting & Learning

Analytics: (8 CSU & 2 UC campuses)

2. Blackboard Analytics for Learn (3 campuses)

3. LMS-agnostic campus surveys

Investigating additional pilot with LMS-agnostic tool to move beyond “clickometry” into social network analysis, discourse analysis, etc.

Raises question for MOOC research: relationship between student intent/motivation, student characteristics/leading indicators, MOOC use, and achievement


Data Dashboar

d

ERS Data

CCA Data

LMS Data

Other Data

Sources

Data Dashboard


Feedback? Questions?

John Whitmer [email protected]

Monograph @ http:www.johnwhitmer.net

Twitter: johncwhitmer

Desdemona [email protected]

mailto:[email protected]

http://bit.ly/15ijySP



improving student achievement with new approaches to data

Education

student lms use

aggregated data slides

larger slides

context slides

trend lms use

student achievement

csu achievement gap

lms use variablesstudent