open academic early alert & risk assessment ap presentation

P r e s e n t e d b y

Members of the Apereo LAI Community

March 18, 2015Sandeep Jayaprakash, Marist

Gary Gilbert, Unicon

OPEN-SOURCE ACADEMIC EARLY ALERT &

RISK ASSESSMENT API

Presenters

Sandeep JayaprakashLearning Analytics Specialist, Marist College

Gary GilbertSoftware Architect, UniconIntegrations & Analytics

Agenda

Marist early Alert framework

Open Learning Analytics vision

Learning Analytics Processor

Demo

Discussion

OAAI: Overview and Impact

EDUCAUSE Next

Generation Learning

Challenges (NGLC)

Funded by Bill and

Melinda Gates Foundations

$250,000 over a 15 month period

Goal: Leverage Big Data concepts to create an

open-source academic early alert system and

research “scaling factors”

OAAI: Overview and Impact

Build learning analytics-based early alert system

Sakai Collaboration and Learning Environment

Secure data capture process for extracting LMS data

Pentaho Business Intelligence Suite

Open-source data mining, integration, analysis & reporting

OAAI Predictive Model released under open license

Predictive Modeling Markup Language

Researching learning analytics scaling factors

How “portable” are predictive models?

What intervention strategies are most effective?

Student Aptitude Data

(SATs, current GPA, etc.)

Student Demographic

Data (Age, gender, etc.)

Sakai Event Log Data

Sakai Gradebook Data

Predictive

Model

Scoring

Identifies students

“at risk” to not

complete course

SIS

Dat

aLM

S D

ata

OAAI Early Alert System Overview

Intervention Deployed

“Awareness” or Online

Academic Support

Environment (OASE)

“Creating an Open Academic Early Alert System”

Model DevelopedUsing Historical Data

Step #1: Developed

model using historical

data

Academic Alert

Report (AAR)

Predictors of

Student Risk

Some predictors

were discarded if

not enough data

was available.

LMS predictors were

measured relative

to course averages.

OAAI Predictive Process

Research Design

Deployed OAAI system to 2200 students across four

institutions

Two Community Colleges

Two Historically Black Colleges and Universities

Design > One instructor teaching 3 sections

One section was control, other 2 were treatment groups

Each instructor received an AAR three times during

the semester:

Intervals were 25%, 50% and 75% into the semester

Prediction Results

Spring ’12 Portability Findings

Fall ’12 Portability Findings

Conclusion

1. Predictive models

are more “portable”

than anticipated.

2. It is possible to

create generic

models that are

then “tuned” for use

at specific types of

institutions.

Intervention Research Findings Final Course Grades

Analysis showed a

statistically significant

positive impact on final

course grades

No difference between

treatment groups

Saw larger impact in

spring than fall

Similar trend amount

low income students

50

60

70

80

90

100

Awareness OASE Control

Fin

al G

rad

e (%

)

Mean Final Grade for "at Risk" Students

Intervention Research Findings Content Mastery

Student in intervention

groups were statistically

more likely to “master

the content” than those

in controls.

Content Mastery = Grade

of C or better

Similar for low income

students.

0

200

400

600

800

1000

Yes No Yes No

Content Mastery for "at Risk" Students

Control Intervention

Freq

uen

cy

Instructor Feedback

"Not only did this project directly assist my students by guiding

students to resources to help them succeed, but as an instructor,

it changed my pedagogy; I became more vigilant about

reaching out to individual students and providing them with

outlets to master necessary skills.

P.S. I have to say that this semester, I received the highest

volume of unsolicited positive feedback from students, who

reported that they felt I provided them exceptional individual

attention!

JAYAPRAKASH, S . M. , MOODY, E . W. , LAURÍA, E . J . ,

REGAN, J . R . , & BARON, J . D . (2014) . EARLY ALERT OF

ACADEMICALLY AT -R ISK STUDENTS : AN OPEN SOURCE

ANALYT ICS IN I T IAT IVE . JOURNAL OF LEARNING

ANALYT ICS , 1 (1) , 6 -47 .

More Research Findings…

Strategic Vision: Open Learning

Analytics PlatformCollectionStandards-based data capture from any potential source using Experience API and/or IMS Caliper/Senor API

StorageSingle repository for all learning-related data using Learning Record Store (LRS) standard.

AnalysisFlexible Learning Analytics Processor (LAP) that can handle data mining, data processing (ETL), predictive model scoring and reporting.

CommunicationDashboard technology for displaying LAP output.

ActionLAP output can be fed into other systems to trigger alerts, etc.

Technology Stack

Learning Analytics Processor (LAP)

JAVA-based web application

Maven for builds

Temporary Storage - H2 in-memory database

Persistence Storage - MySQL

Predictive Model Mark-up Language (PMML)

OAAI Early Alert Pipeline

Pentaho Kettle – Data Integration & ETL

Pentaho WEKA – Data Mining & Predictive Modelling

High-Level Workflow

Sakai

Admin

tool

activities.csv

grades.csv

Learning Analytics Processor (LAP)Student ID,

Course ID,

Risk Rating

Demographics

from SIS

Go!

grabs files

OAAI XML

Kettle pipeline

applies model

outputs results

..

.

.

------------------ EXTRACT -------- TRANSFORM ------- LOAD ---------

RESTful API

LAP Pipeline Architecture

Features

Key pieces of the LAP architecture

Input source management

Data storage – temporary & persistent

Configuration manager

Pipeline processor

Output results management

Extensibility

Supports multiple pipelines

Supports varied pipeline platforms

Demo Overview

● Three core components of a

collection of open source

applications and services that

represent the “Analytics Diamond”

● Can be used individually or

collectively

● Work with a shared infrastructure

and data model

Technologies:

• AngularJS

• Spring-Boot

• Pluggable Datastores

(redis, elasticsearch, mongodb)

OpenLRS

Learning

Analytics

Processor

SakaiOpen

Dashboard

xAPI

LTI

API

API

Demo

Early Alert Insights – Open Dash

Questions?

APEREO LEARNING ANALYTICS INITIATIVE COMMUNITY

• Accelerate the operationalization of Learning Analytics software and frameworks

• Support the validation of analytics pilots across institutions

• Work together so as to avoid duplication

analytics-coordinator@apereo.org

Josh Baronjosh.baron@marist.edu

Sandeep Jayaprakashsandeep.jayaprakash1@

marist.edu

Gary Gilbertggilbert@unicon.net

Appendix

Early Alert - Kettle ETL Flows

WEKA Predictive Modelling Flows

Learning Analytics Processor