Using real-time dashboards to improve student engagement in

virtual learning environmentsRobert Bodily, Steven WoodBrigham Young University

Research questionsDashboard DevelopmentR1: What technical requirements are needed for an online learning system to collect and provide students with personalized information in a student dashboard?R2: What functionality do students want in a dashboard and how should it be visually represented?Data Mining AnalysisR3: What is predictive of student success in a first-year general chemistry course?R4: Can we develop an early alert warning system using clickstream data?

R1: Technical Requirements

Challenges with current LMS

• A lot of learning is not occurring within a Learning Management System (LMS)

• Interoperability standards

• No access to real-time data• Canvas data (1 day old)• API (rate limiting factors)

• Do not track enough data• No information on how students interact with a page

Our analytics system

LTI (Learning Tools Interoperability)Single sign-on system for learning applications and learning management systems

xAPI (Experience API/Tincan API) Data format specification for data management interoperability

LRS (Learning Record StoreDatabase that stores xAPI statements and provides real-time data access

R2: Functionality

Student Dashboard

Challenges:

Low student use

Initially not useful for students

Students had too many things to do already

Low engagement with feedback

Instructor dashboard

R3: Predictive elements of student success

Context

• Class• First year chemistry course• Blended – class 3x per week,

• Resources• 150 videos (avg. 2 min long, supplemental resources)• 15 weekly quizzes (unlimited question attempts)

• Participants• 200 students (online interactions)• 96 students took the self-report resource use survey

Data collected• Quiz

• Confidence in answer (just a guess, pretty sure, very sure)• Time spent on quiz• Correct/incorrect• Number of attempts per question• Leave tab (still open, but inactive), come back to tab (active again)

• Video• Play, pause, skip forward/backward, change play rate, change volume,

• Dashboard• Number of times students follow recommendations given in dashboard• Number of clicks within the dashboard

What course elements are predictive of student success?

Variable Beta P-valueOnline homework score 0.366 0.000In-class IClicker scores 0.154 0.024# of attempts/question -0.411 0.000Amount of question navigation -0.206 0.040# of online activity sessions -0.195 0.020

Variable Beta P-valueRead the textbook 2.443 0.059Ask professor questions in class 7.363 0.000Watch Khan Academy -2.738 0.051Use the internet -3.199 0.010Skip recitation -4.820 0.041

Model 1 – regressing online interaction data on final exam score.

Model 2 – regressing self-report resource use on final exam score.

R4: Early alert warning system

Develop an early course prediction of student achievement

Online student interaction data Online student interaction data AND exam scores

There is significant improvement in both models until week 3 or 4, so that seems to be a good time to make predictions for instructors and students.

Thank you! Questions?