Download - Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course
![Page 1: Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course](https://reader036.vdocuments.net/reader036/viewer/2022062811/568160a7550346895dcfc977/html5/thumbnails/1.jpg)
John Whitmer, Ed.D.Updated: February 19, 2013
Using Learner Analytics to Understand Student Achievement in
a Large Enrollment Hybrid Course
![Page 2: Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course](https://reader036.vdocuments.net/reader036/viewer/2022062811/568160a7550346895dcfc977/html5/thumbnails/2.jpg)
Outline1. Context
2. Methods & Tools
3. Findings
4. Conclusions & Next Steps
![Page 3: Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course](https://reader036.vdocuments.net/reader036/viewer/2022062811/568160a7550346895dcfc977/html5/thumbnails/3.jpg)
1. CONTEXT
![Page 4: Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course](https://reader036.vdocuments.net/reader036/viewer/2022062811/568160a7550346895dcfc977/html5/thumbnails/4.jpg)
Case Study: Intro to Religious Studies• Undergraduate, introductory, high demand
• Redesigned to hybrid delivery format through “academy eLearning program”
• Enrollment: 373 students (54% increase on largest section)
• Highest LMS (Vista) usage entire campus Fall 2010 (>250k hits)
• Bimodal outcomes:• 10% increase on final exam• 7% & 11% increase in DWF
• Why? Can’t tell with aggregated data
54 F’s
![Page 5: Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course](https://reader036.vdocuments.net/reader036/viewer/2022062811/568160a7550346895dcfc977/html5/thumbnails/5.jpg)
Founded in 1887
15,257 FTES, 95% from California, serves 12 counties
Primarily residential, undergraduate teaching college
Campus in California State University system (23 colleges, 44,000 faculty and staff, 437,000 students)
![Page 6: Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course](https://reader036.vdocuments.net/reader036/viewer/2022062811/568160a7550346895dcfc977/html5/thumbnails/6.jpg)
CSU Budget Proposed Increase!
Source: CSU Chancellor’s Officehttp://bit.ly/X7LYeK
![Page 7: Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course](https://reader036.vdocuments.net/reader036/viewer/2022062811/568160a7550346895dcfc977/html5/thumbnails/7.jpg)
Driving Conceptual Questions1. How is student LMS use related to academic
achievement in a single course section?
2. How does that finding compare to the relationship of achievement with traditional student characteristic variables?
3. How are these relationships different for “at-risk” students (URM & Pell-eligible)?
4. What data sources, variables and methods are most useful to answer these questions?
![Page 8: Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course](https://reader036.vdocuments.net/reader036/viewer/2022062811/568160a7550346895dcfc977/html5/thumbnails/8.jpg)
Gender Freq. PercentUniversity Average Difference
Female 231 62% 51% 11%Male 142 38% 48% -10%
Age 0% 17 22 6% 18-21 302 81% 22-30 22 6% 31+ 1 0%
Under-represented Minority
No 264 71% 73% -2%Yes 109 29% 27% 2%
Pell-eligible Freq. Percent No 210 56% Yes 163 44%
First Attend College Freq. No 268 72% Yes 105 28%
Enrollment Status Freq. Continuing Student 217 58% Transfer 17 5% First-Time Student 139 37%
![Page 9: Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course](https://reader036.vdocuments.net/reader036/viewer/2022062811/568160a7550346895dcfc977/html5/thumbnails/9.jpg)
2. METHODS & TOOLS
![Page 10: Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course](https://reader036.vdocuments.net/reader036/viewer/2022062811/568160a7550346895dcfc977/html5/thumbnails/10.jpg)
Methods at a Glance Data Sources: 1) LMS logfiles, 2) SIS data,
3) Course data
Process1. Clean/filter/transform/reduce data (70% effort)2. Descriptive / exploratory analysis (20% effort)3. Statistical analysis (10% effort)
Factor analysis Correlation single variables Regression multiple variables; partial & complete
![Page 11: Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course](https://reader036.vdocuments.net/reader036/viewer/2022062811/568160a7550346895dcfc977/html5/thumbnails/11.jpg)
Tools Used
App Function
Excel Early data exploration; simple sorting; tables for print/publication
Tableau Complex data summaries and explorations; complex charts; presentation chartsFinal/formal descriptive data; statistical analysis; some charts (scatterplots)Statistical analysis (factor analysis)
![Page 12: Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course](https://reader036.vdocuments.net/reader036/viewer/2022062811/568160a7550346895dcfc977/html5/thumbnails/12.jpg)
Variables
![Page 13: Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course](https://reader036.vdocuments.net/reader036/viewer/2022062811/568160a7550346895dcfc977/html5/thumbnails/13.jpg)
Missing Data On Critical Indicators
![Page 14: Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course](https://reader036.vdocuments.net/reader036/viewer/2022062811/568160a7550346895dcfc977/html5/thumbnails/14.jpg)
Final data set: 72,000 records (-73%)
![Page 15: Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course](https://reader036.vdocuments.net/reader036/viewer/2022062811/568160a7550346895dcfc977/html5/thumbnails/15.jpg)
LMS Use Consistent across Categories
Factor Analysis of LMS Use Categories
![Page 16: Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course](https://reader036.vdocuments.net/reader036/viewer/2022062811/568160a7550346895dcfc977/html5/thumbnails/16.jpg)
3. FINDINGS
![Page 17: Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course](https://reader036.vdocuments.net/reader036/viewer/2022062811/568160a7550346895dcfc977/html5/thumbnails/17.jpg)
Clear Trend: Grade w/Mean LMS Hits
![Page 18: Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course](https://reader036.vdocuments.net/reader036/viewer/2022062811/568160a7550346895dcfc977/html5/thumbnails/18.jpg)
Question 1 Results: Correlation LMS Use w/Final Grade
Scatterplot of Assessment Activity
Hits vs. Course Grade
![Page 19: Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course](https://reader036.vdocuments.net/reader036/viewer/2022062811/568160a7550346895dcfc977/html5/thumbnails/19.jpg)
Question 2 Results: Correlation: Student Char. w/Final Grade
Scatterplot of HS GPA vs. Course
Grade
![Page 20: Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course](https://reader036.vdocuments.net/reader036/viewer/2022062811/568160a7550346895dcfc977/html5/thumbnails/20.jpg)
Question 2 Results: Correlation: Student Char. w/Final Grade
![Page 21: Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course](https://reader036.vdocuments.net/reader036/viewer/2022062811/568160a7550346895dcfc977/html5/thumbnails/21.jpg)
Conclusion: LMS Use Variables better Predictors than Student Characteristics
LMS Use
Variables
18% Average(r = 0.35–0.48)
Explanation of change in final grade
Student Characteristic
Variables
4% Average(r = -0.11–0.31)
Explanation of change in final grade
>
![Page 22: Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course](https://reader036.vdocuments.net/reader036/viewer/2022062811/568160a7550346895dcfc977/html5/thumbnails/22.jpg)
Smallest LMS Use Variable
(Administrative Activities)
r = 0.35
Largest Student
Characteristic
(HS GPA)
r = 0.31
>
![Page 23: Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course](https://reader036.vdocuments.net/reader036/viewer/2022062811/568160a7550346895dcfc977/html5/thumbnails/23.jpg)
Combined Variables Regression Final Grade by LMS Use & Student Characteristic Variables
LMS Use
Variables
25% (r2=0.25)
Explanation of change in final grade
Student Characteristic
Variables
+10%(r2=0.35)
Explanation of change in final grade
>
![Page 24: Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course](https://reader036.vdocuments.net/reader036/viewer/2022062811/568160a7550346895dcfc977/html5/thumbnails/24.jpg)
Question 3 Results:Regression by “At Risk” Population Subsamples
![Page 25: Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course](https://reader036.vdocuments.net/reader036/viewer/2022062811/568160a7550346895dcfc977/html5/thumbnails/25.jpg)
At-Risk Students: “Over-Working Gap”
25
![Page 26: Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course](https://reader036.vdocuments.net/reader036/viewer/2022062811/568160a7550346895dcfc977/html5/thumbnails/26.jpg)
Activities by Pell and Gradegrade / pelleligible
A B+ C C-
Pell-Eligible Not Pell-Eligible Pell-Eligible Not Pell-Eligible Pell-Eligible Not Pell-Eligible Pell-Eligible Not Pell-Eligible0K
5K
10K
15K
20K
25K
30K
35K
Value
Content
Content
Engage
Engage
Assess
Assess
Admin
Admin
Content
Content
Engage
Engage
Assess
Assess
Admin
Content
Content
Engage
Engage
Assess
Assess
Content
Content Engage
Engage
Assess
Assess
Admin
Admin
Measure NamesAdmin
Assess
Engage
Content
Extra effort in content-related activities
![Page 27: Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course](https://reader036.vdocuments.net/reader036/viewer/2022062811/568160a7550346895dcfc977/html5/thumbnails/27.jpg)
Previous Studies Relating LMS Use to Course Grade
![Page 28: Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course](https://reader036.vdocuments.net/reader036/viewer/2022062811/568160a7550346895dcfc977/html5/thumbnails/28.jpg)
4. CONCLUSIONS & NEXT STEPS
![Page 29: Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course](https://reader036.vdocuments.net/reader036/viewer/2022062811/568160a7550346895dcfc977/html5/thumbnails/29.jpg)
Conclusions1. At the course level, LMS use better predictor of
academic achievement than student demographics (what do, not who are).
2. Small strength magnitude of complete model demonstrates relevance of data, but suggests that better methods could produce stronger results.
3. LMS data requires extensive filtering to be useful; student variables need pre-screening for missing data.
![Page 30: Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course](https://reader036.vdocuments.net/reader036/viewer/2022062811/568160a7550346895dcfc977/html5/thumbnails/30.jpg)
More Conclusions4. LMS use frequency is a proxy for effort. Not a
very complex indicator.
5. Student demographic measures need revision for utility in Postmodern era (importance to student, more frequent sampling, etc.).
6. LMS effectiveness for at-risk students may be caused by non-technical barriers. Need additional research!
![Page 31: Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course](https://reader036.vdocuments.net/reader036/viewer/2022062811/568160a7550346895dcfc977/html5/thumbnails/31.jpg)
Ideas & FeedbackPotential for improved LMS analysis methods: social learning activity patterns discourse content analysis time series analysis
Group students by broader identity, with unique variables: Continuing student (Current college GPA, URM, etc. First-time freshman (HS GPA, SAT/Act, etc)
![Page 32: Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course](https://reader036.vdocuments.net/reader036/viewer/2022062811/568160a7550346895dcfc977/html5/thumbnails/32.jpg)
Feedback? Questions?
John Whitmer [email protected]
Slideshttp://slidesha.re/15iokzE
Complete monographhttp://bit.ly/15ijySP
Twitter: johncwhitmer