John Whitmer, Ed.D.Associate Director, Academic Technology Services

California State University, Office of the Chancellor

Society for Learning Analytics Research | LAK 2013 Case StudyFebruary 19, 2013

Using Learner Analytics to Understand Student Achievement in

a Large Enrollment Hybrid Courseslides posted:

Outline

1. Context

2. Methods & Tools

3. Findings

4. Conclusions & Next Steps

1. CONTEXT

Founded in 1887

15,257 FTES, 95% from California, serves 12 counties

Primarily residential, undergraduate teaching college

Campus in California State University system (23 colleges, 44,000 faculty and staff, 437,000 students)

CSU Budget Proposed Increase!

Source: CSU Chancellor’s Officehttp://bit.ly/X7LYeK

Case Study: Intro to Religious Studies

• Undergraduate, introductory, high demand

• Redesigned to hybrid delivery format through “academy eLearning program”

• Enrollment: 373 students (54% increase on largest section)

• Highest LMS (Vista) usage entire campus Fall 2010 (>250k hits)

• Bimodal outcomes:• 10% increase on final exam• 7% & 11% increase in DWF

• Why? Can’t tell with aggregated data

54 F’s

Driving Conceptual Questions

1. How is student LMS use related to academic achievement in a single course section?

2. How does that finding compare to the relationship of achievement with traditional student characteristic variables?

3. How are these relationships different for “at-risk” students (URM & Pell-eligible)?

4. What data sources, variables and methods are most useful to answer these questions?

Gender Freq. PercentUniversity Average Difference

Female 231 62% 51% 11%Male 142 38% 48% -10%

Age 0% 17 22 6%   18-21 302 81%   22-30 22 6%   31+ 1 0%   

Under-represented Minority  

No 264 71% 73% -2%Yes 109 29% 27% 2%

Pell-eligible Freq. Percent    No 210 56%   Yes 163 44%   

First Attend College Freq.      No 268 72%   Yes 105 28%   

Enrollment Status Freq.      Continuing Student 217 58%   Transfer 17 5%   First-Time Student 139 37%   

2. METHODS & TOOLS

Methods at a Glance

Data Sources: 1) LMS logfiles, 2) SIS data, 3) Course data

Process1. Clean/filter/transform/reduce data (70% effort)

2. Descriptive / exploratory analysis (20% effort)

3. Statistical analysis (10% effort) Factor analysis Correlation single variables Regression multiple variables; partial & complete

Tools Used

App Function

Excel Early data exploration; simple sorting; tables for print/publication

Tableau Complex data summaries and explorations; complex charts; presentation charts

Final/formal descriptive data; statistical analysis; some charts (scatterplots)

Statistical analysis (factor analysis)

Variables

Missing Data On Critical Indicators

Final data set: 72,000 records (-73%)

LMS Use Consistent across Categories

Factor Analysis of LMS Use Categories

3. FINDINGS

Clear Trend: Grade w/Mean LMS Hits

Question 1 Results: Correlation LMS Use w/Final Grade

Scatterplot of Assessment Activity

Hits vs. Course Grade

Question 2 Results: Correlation: Student Char. w/Final Grade

Scatterplot of HS GPA vs.

Course Grade

Conclusion: LMS Use Variables better Predictors than Student Characteristics

LMS Use

Variables

18% Average(r = 0.35–0.48)

Explanation of change in final grade

Student Characteristic

Variables

4% Average(r = -0.11–0.31)

Explanation of change in final grade

>

Smallest LMS Use Variable

(Administrative Activities)

r = 0.35

Largest Student

Characteristic

(HS GPA)

r = 0.31

>

Combined Variables Regression Final Grade by LMS Use & Student Characteristic Variables

LMS Use

Variables

25% (r2=0.25)

Explanation of change in final grade

Student Characteristic

Variables

+10%(r2=0.35)

Explanation of change in final grade

>

Question 3 Results:Regression by “At Risk” Population Subsamples

At-Risk Students: “Over-Working Gap”

24

Activities by Pell and Gradegrade / pelleligible

A B+ C C-

Pell-Eligible Not Pell-Eligible Pell-Eligible Not Pell-Eligible Pell-Eligible Not Pell-Eligible Pell-Eligible Not Pell-Eligible

0K

5K

10K

15K

20K

25K

30K

35K

Value

Content

Content

Engage

Engage

Assess

Assess

Admin

Admin

Content

Content

Engage

Engage

Assess

Assess

Admin

Content

Content

Engage

Engage

Assess

Assess

Content

Content Engage

Engage

Assess

Assess

Admin

Admin

Measure Names

Admin

Assess

Engage

Content

Extra effort in content-related activities

4. CONCLUSIONS & NEXT STEPS

Conclusions

1. At the course level, LMS use better predictor of academic achievement than student demographics (what do, not who are).

2. Small strength magnitude of complete model demonstrates relevance of data, but suggests that better methods could produce stronger results.

3. LMS data requires extensive filtering to be useful; student variables need pre-screening for missing data.

More Conclusions

4. LMS use frequency is a proxy for effort. Not a very complex indicator.

5. Student demographic measures need revision for utility in Postmodern era (importance to student, more frequent sampling, etc.).

6. LMS effectiveness for at-risk students may be caused by non-technical barriers. Need additional research!

Ideas & Feedback

Potential for improved LMS analysis methods: social learning activity patterns discourse content analysis time series analysis

Group students by broader identity, with unique variables: Continuing student (Current college GPA, URM, etc. First-time freshman (HS GPA, SAT/Act, etc)

Feedback? Questions?

John Whitmer jwhitmer@calstate.edu

Slides

Complete monographhttp://bit.ly/15ijySP

Twitter: johncwhitmer