what data from 3 million learners can tell us about effective course design
TRANSCRIPT
What data from 3 million learners can tell us about effective course designJohn Whitmer, Ed.D.Director, Analytics & [email protected] | @johncwhitmer
22
Meta- questions driving our Learning Analytics research @ Blackboard
1. How do students & teachers use our platforms? How is this use related to student achievement? [or satisfaction, or risk, or …]
3. How can we integrate these findings into features/functionality that apply to the broad spectrum of ways people use our platforms?
2. Do these findings apply equally to students ‘at promise’ due to their academic achievement or background characteristics? (e.g. race, class, family education, geography)
33
Outline
1. Positioning Learning Analytics
2. Research Findings
3. Q & A
4
1. Positioning Learning Analytics
55
66Economist. (2010, 11/4/2010). Augmented business: Smart systems will disrupt lots of industries, and perhaps the entire economy. The Economist.
200MB of data emissions annually
77
Logged into course within 24 hours
Failed first exam
Interacts frequently in discussion boards
No declared majorHasn’t taken college-level math
8
“ ...measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs.”
Learning and Knowledge Analytics Conference, 2011
What is learning analytics
9
Strong interest by faculty and students
1010
Blackboard’s Learning Data Footprint (2015 #’s)
1.6M Unique Courses
40M Course Content Items
4M Unique Students
Blackboard Learn = ¼ total data
775M LMS Sessions
1111
Commitment to Privacy & Openness
• Analyze data records that are not only removed of PII, but de-personalized (individual & institutional levels)
• Share results and open discussion procedures for analysis to inform broader educational community
• Respect territorial jurisdictions and safe harbor provisions
12
2. Research Findings
13
Why this is hard?
• LMS data is messy
• Adoption is hugely varied
14
Same model doesn't work for all courses
• 1.2M students
• 34,519 courses
• 788 institutions
• Overall effect size < 1%
15
But strong effect in some courses (n=7,648, 22%)
16
The Effectiveness of Blackboard Learn Tool Usage
17
Purpose of InvestigationTo determine the most important tools in Bb Learn, by observing:
– Tools that are used the most (in minutes, for instance)
– Tools that have strongest relationship with final grade
– Tools that are ‘underused’ the most (by learners & instructors); tools that have the greatest potential to improve learning outcomes
Allows us to see which tools educate students, and are therefore useful
Reinforce the educational impact of the Blackboard Learn platform
18
Data Filtering
Filters decreased the number of students analyzed from 3.37 million users in 70,000 courses from 927 institutions to 601,544 users (17% of total) in 18,810 courses (26.8% of total) from 663 institutions (71.5% of total)
Class Sizebetween 10 and 500 students
Activity Ratesover 1 hour online as a course
average
Grade Distribution average grade between 40% & 95%
19
Finding: Tool Use & GradeTool use and Final Grade do not have a linear relationship; there is a diminishing marginal effect of tool use on Final Grade
Interpretations
• Students absent from course activity are at greatest risk of low achievement.
• The first time you read/see a PowerPoint presentation, you learn a lot, but the second time you read/see it, you learn less.
• Getting from a 90% to a 95% requires more effort than getting from a 60% to a 65%.
20
Finding: Tool Use & GradeTool use and Final Grade do not have a linear relationship; there is a diminishing marginal effect of tool use on Final Grade
Interpretations
• Students absent from course activity are at greatest risk of low achievement.
• The first time you read/see a PowerPoint presentation, you learn a lot, but the second time you read/see it, you learn less.
• Getting from a 90% to a 95% requires more effort than getting from a 60% to a 65%.
Log transformation shows stronger trend
21
Investigation Achievement by Specific Tools UsedAnalysis Steps
• Identify most frequently used tools
• Separate tool use into no use + quartiles
• Divide students into 3 groups by course grade• High (80+)• Passing (60-79)• Low/Failing (0-59)
22
Finding: MyGradesAt every level, probability of higher grade increases with increased use. Causal? Probably not. Good indicator? Absolutely.
23
Finding: Course contentsMore is not always better. Large jump none to some; then no relationship
24
Finding: Assessments/AssignmentsStudents above mean have lower likelihood of achieving a high grade than students below the mean
25
Next Steps & Discussion
26
Next Steps in Product & Research• Refine Ultra “Learning Analytics” triggers for low/high achievement; focus on
getting started, not achieving “top of class” in activity.
• Explore data points beyond time on task; semantic analysis, writing level analysis, other more rich data points
• Investigate course design structures and patterns in how teachers create course experiences using Learn
• Collaborate with institutions on research to consider alternative measures of success besides course final grade (course evaluations, grades in subsequent courses)
